Nov 03 00:04:35.362222 ip-10-0-133-253 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:04:35.796836 ip-10-0-133-253 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:35.796836 ip-10-0-133-253 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:04:35.796836 ip-10-0-133-253 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:35.796836 ip-10-0-133-253 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:35.801110 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801023    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.801110 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801109    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.801110 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801113    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801116    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801119    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801122    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801125    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801129    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801131    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801134    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801137    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801139    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801142    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801145    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801148    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801150    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801153    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801157    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801161    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801165    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801179    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.801307 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801182    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801191    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801194    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801197    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801199    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801202    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801205    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801207    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801210    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801213    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801216    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801218    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801221    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801224    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801226    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801230    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801232    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801235    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801238    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801240    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.801755 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801243    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801246    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801248    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801251    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801253    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801256    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801268    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801271    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801273    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801276    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801278    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801280    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801286    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801289    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801291    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801294    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801296    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801299    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.802265 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801302    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801305    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801308    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801311    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801314    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801316    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801319    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801322    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801325    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801327    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801330    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801333    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801335    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801338    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801342    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801346    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801349    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801351    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801354    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801356    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.802740 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801365    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801368    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801370    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801373    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801375    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801378    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801381    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801383    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801386    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801388    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801391    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801394    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801397    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801400    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801403    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801405    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801408    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801411    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801413    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801416    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.803261 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801418    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801421    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801424    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801426    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.801430    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802314    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802323    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802326    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802329    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802332    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802335    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802337    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802340    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802344    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802347    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802350    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802352    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802355    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802358    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802361    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.803723 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802363    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802366    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802369    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802372    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802374    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802377    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802380    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802382    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802386    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802388    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802391    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802394    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802397    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802399    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802402    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802404    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802407    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802409    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802412    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.804243 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802414    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802416    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802419    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802422    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802426    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802430    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802433    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802437    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802440    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802442    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802445    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802447    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802450    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802452    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802455    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802458    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802460    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802462    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802465    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802467    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.804700 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802470    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802473    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802475    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802478    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802480    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802483    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802485    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802488    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802490    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802492    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802496    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802498    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802501    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802503    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802506    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802508    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802511    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802514    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802516    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.805228 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802519    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802522    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802525    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802528    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802531    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802534    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802536    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802539    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802547    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802550    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802552    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802555    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802558    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802560    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802563    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802566    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802570    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802574    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802577    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802580    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.805663 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802582    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802585    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802587    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802590    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802593    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802595    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802598    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802600    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802603    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802605    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802608    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802610    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.802615    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803140    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803146    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803149    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803152    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803155    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803157    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803160    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.806144 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803162    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803165    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803168    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803171    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803174    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803176    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803179    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803182    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803185    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803188    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803190    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803193    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803196    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803198    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803201    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803203    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803206    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803208    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803211    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.806611 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803213    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803216    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803218    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803220    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803223    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803228    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803230    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803234    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803237    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803241    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803245    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803248    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803251    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803254    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803257    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803260    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803263    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803266    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803268    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.807143 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803271    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803274    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803277    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803279    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803282    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803284    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803287    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803290    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803292    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803295    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803297    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803299    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803302    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803304    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803307    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803309    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803312    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803314    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803317    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803320    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.807650 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803322    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803326    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803328    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803331    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803334    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803336    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803339    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803341    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803343    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803346    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803348    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803350    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803355    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803358    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803362    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803364    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803367    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803369    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803372    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803374    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.808201 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803377    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803379    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803382    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803384    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803387    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803389    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803392    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803394    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803397    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803399    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803402    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803404    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803406    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803409    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803412    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803415    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803417    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803419    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803422    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803426    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.808721 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.803429    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804482    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804491    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804502    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804507    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804512    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804515    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804522    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804526    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804530    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804533    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804537    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804540    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804543    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804547    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804550    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804553    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804556    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804559    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804562    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804565    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804568    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804572    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804576    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:04:35.809270 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804580    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804583    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804586    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804590    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804593    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804596    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804599    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804604    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804607    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804612    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804615    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804618    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804621    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804624    2579 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804627    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804632    2579 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804635    2579 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804638    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804642    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804645    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804649    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804652    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804656    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804659    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804662    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:04:35.809880 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804665    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804668    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804671    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804674    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804677    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804680    2579 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804684    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804687    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804690    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804694    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804697    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804701    2579 flags.go:64] FLAG: --help="false"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804704    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-253.ec2.internal"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804708    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804713    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804716    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804719    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804723    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804726    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804729    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804732    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804735    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804738    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804741    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:04:35.810555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804744    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804747    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804750    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804753    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804756    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804760    2579 flags.go:64] FLAG: --lock-file=""
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804763    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804766    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804769    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804775    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804778    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804781    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804784    2579 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804787    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804791    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804794    2579 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804797    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804801    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804804    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804808    2579 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804812    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804815    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804819    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804822    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804825    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:04:35.811194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804828    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804831    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804839    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804842    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804845    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804849    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804851    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804854    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804858    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804861    2579 flags.go:64] FLAG: --port="10250"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804864    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804867    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b9fca6005c2918e4"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804871    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804875    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804878    2579 flags.go:64] FLAG: --register-node="true"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804881    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804885    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804888    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804891    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804894    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804897    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804901    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804904    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804907    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804909    2579 flags.go:64] FLAG: --runonce="false"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804925    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:04:35.811790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804929    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804932    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804935    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804938    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804941    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804945    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804948    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804951    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804954    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804957    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804960    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804963    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804966    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804969    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804974    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804977    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804980    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804987    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804992    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804995    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.804997    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805001    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805004    2579 flags.go:64] FLAG: --v="2"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805008    2579 flags.go:64] FLAG: --version="false"
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805012    2579 flags.go:64] FLAG: --vmodule=""
Nov 03 00:04:35.812440 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805016    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805019    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805153    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805157    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805160    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805164    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805175    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805179    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805182    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805184    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805188    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805190    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805193    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805196    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805198    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805201    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805203    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805206    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805209    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805211    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.813073 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805213    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805216    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805219    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805221    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805224    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805229    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805232    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805235    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805237    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805240    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805243    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805245    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805248    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805251    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805254    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805256    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805259    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805261    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805264    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805266    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.813584 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805269    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805271    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805274    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805277    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805279    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805282    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805284    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805287    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805289    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805291    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805294    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805296    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805299    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805301    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805304    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805306    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805309    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805312    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805315    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.814086 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805318    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805321    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805323    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805326    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805328    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805331    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805333    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805336    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805338    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805341    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805343    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805346    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805350    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805353    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805355    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805358    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805361    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805364    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805366    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805369    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.814563 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805371    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805374    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805376    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805379    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805382    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805384    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805387    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805389    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805393    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805397    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805401    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805404    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805406    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805408    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805411    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805414    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805416    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805419    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805422    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.815046 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805424    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805427    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805431    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805434    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805436    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805440    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805443    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805446    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805449    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805451    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.805478    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805513    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805518    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805521    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805524    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805527    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.815490 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805529    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805532    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805535    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805537    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805540    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805542    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805545    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805549    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805551    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805554    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805557    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805559    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805562    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805564    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805567    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805570    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805572    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805575    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805577    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.815873 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805580    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805583    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805585    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805589    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805592    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805594    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805597    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805599    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805602    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805605    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805608    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805610    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805613    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805615    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805618    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805620    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805623    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805626    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805629    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.816353 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805631    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805635    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805638    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805640    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805643    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805645    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805648    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805650    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805653    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805655    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805658    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805660    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805663    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805666    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805668    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805673    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805678    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805681    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805684    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.816841 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805687    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805690    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805692    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805695    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805698    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805701    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805704    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805707    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805709    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805714    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805718    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805720    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805723    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805726    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805730    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805732    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805735    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805737    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805740    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805743    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.817567 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805745    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805748    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805750    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805753    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805755    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805758    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805760    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805763    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805766    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805770    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805772    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805775    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805777    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805781    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805783    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805786    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805789    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805792    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805794    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805797    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.818347 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805800    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805802    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805805    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.805807    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.813935    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.813953    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.813969    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.813974    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814033    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814039    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814043    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814046    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814049    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814052    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814054    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814057    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814060    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814062    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814065    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.818867 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814068    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814071    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814074    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814077    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814079    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814082    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814084    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814087    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814089    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814092    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814095    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814098    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814100    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814103    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814106    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814108    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814111    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814113    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814116    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.819387 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814119    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814122    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814125    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814127    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814130    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814132    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814135    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814137    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814140    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814142    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814145    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814147    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814150    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814152    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814155    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814157    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814160    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814163    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814166    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.819820 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814168    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814170    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814173    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814175    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814178    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814180    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814183    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814185    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814188    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814190    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814193    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814195    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814197    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814200    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814203    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814206    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814208    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814211    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814213    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814216    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.820355 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814218    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814221    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814223    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814225    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814230    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814233    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814236    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814238    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814241    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814244    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814246    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814249    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814251    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814254    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814256    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814259    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814261    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814264    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814267    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.820851 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814269    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814272    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814274    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814276    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814279    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814282    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814284    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814287    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814290    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814292    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814295    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814297    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814299    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814302    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814304    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814307    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814309    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.821315 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814312    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.814339    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814475    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814480    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814484    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814487    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814490    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814493    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814495    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814498    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814501    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814503    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814506    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814509    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814511    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814514    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.821711 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814516    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814519    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814522    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814524    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814527    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814529    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814531    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814534    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814537    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814541    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814544    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814547    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814550    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814553    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814556    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814558    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814561    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814563    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814566    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.822101 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814569    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814571    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814574    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814577    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814580    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814582    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814585    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814587    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814590    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814593    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814595    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814598    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814600    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814603    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814605    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814608    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814610    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814613    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814615    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814617    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.822548 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814620    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814623    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814626    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814628    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814631    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814633    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814636    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814638    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814641    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814643    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814645    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814648    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814651    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814656    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814658    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814661    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814664    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814666    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814669    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.823025 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814671    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814674    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814676    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814679    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814682    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814685    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814687    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814690    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814692    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814695    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814697    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814700    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814702    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814704    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814707    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814710    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814712    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814715    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814718    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814720    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.823463 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814722    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814725    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814727    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814730    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814732    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814735    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814739    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814741    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814744    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814746    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814749    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814751    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814754    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814756    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.814779    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814811    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:35.823929 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814815    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814819    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814822    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814825    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814828    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814831    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814833    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814836    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814839    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814841    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814844    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814846    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814849    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814852    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814855    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814857    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814859    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814862    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814864    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:35.824329 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814867    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814869    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814872    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814876    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814878    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814881    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814883    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814886    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814888    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814891    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814893    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814896    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814899    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814901    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814904    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814906    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814909    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814911    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814926    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814929    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:35.824800 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814931    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814934    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814936    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814939    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814941    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814944    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814947    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814952    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814955    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814958    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814961    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814964    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814966    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814969    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814972    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814976    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814979    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814981    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814984    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814986    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:35.825369 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814989    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814992    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814994    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814997    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.814999    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815002    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815004    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815007    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815009    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815013    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815017    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815020    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815022    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815025    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815028    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815030    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815033    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815035    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815038    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:35.825865 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815041    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815043    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815048    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815051    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815053    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815056    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815059    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815061    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815065    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815068    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815071    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815073    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815076    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815079    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815081    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815084    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815086    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815089    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815091    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815094    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:35.826345 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815096    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815099    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815102    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815105    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815107    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815110    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:35.815112    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.815164    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.819005    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.820112    2579 server.go:1014] "Starting client certificate rotation"
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.820220    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.820264    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.822398    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:04:35.826824 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.823538    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:04:35.835956 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.835937    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:04:35.839055 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.839041    2579 log.go:25] "Validated CRI v1 image API"
Nov 03 00:04:35.839934 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.839890    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:04:35.844294 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.844268    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80e57c27-4e91-4c5b-b31a-0d2221eb3b9c:/dev/nvme0n1p4 a489c620-25f3-4da9-9a71-7b177d94d869:/dev/nvme0n1p3]
Nov 03 00:04:35.844356 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.844293    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 00:04:35.849407 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.849302    2579 manager.go:217] Machine: {Timestamp:2026-11-03 00:04:35.847934969 +0000 UTC m=+0.372650624 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100144 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29efc9ee0af4b493ab04a3f3da813f SystemUUID:ec29efc9-ee0a-f4b4-93ab-04a3f3da813f BootID:c43d4445-44af-4843-b038-ad126b028d21 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d9:80:5f:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d9:80:5f:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:4d:cd:51:e8:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:04:35.849478 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.849410    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:04:35.849513 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.849487    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:04:35.850783 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.850758    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:04:35.850933 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.850784    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:04:35.850976 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.850939    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:04:35.850976 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.850946    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:04:35.850976 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.850963    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:04:35.850976 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.850976    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:04:35.852224 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.852213    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:04:35.857501 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.857488    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:04:35.857635 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.857625    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:04:35.858506 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.858496    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:04:35.858538 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.858515    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:04:35.858538 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.858527    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:04:35.858538 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.858536    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:04:35.858661 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.858548    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:04:35.858999 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.858982    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:04:35.861763 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.861724    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:04:35.870171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.870151    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:04:35.871123 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871109    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:04:35.871194 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871150    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:04:35.871343 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871330    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:04:35.871403 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871352    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:04:35.871403 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871361    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:04:35.871403 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871371    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:04:35.871403 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871381    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:04:35.871403 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871390    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:04:35.872002 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.871991    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:04:35.872061 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.872005    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:04:35.872061 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.872017    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:04:35.872061 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.872028    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:04:35.872061 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.872042    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:04:35.872061 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.872056    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:04:35.873629 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.873615    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:04:35.873701 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.873633    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:04:35.877692 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.877675    2579 server.go:1263] "Started kubelet"
Nov 03 00:04:35.877784 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.877740    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:04:35.877855 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.877810    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:04:35.877907 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.877870    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:04:35.878431 ip-10-0-133-253 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:04:35.879538 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.879487    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:04:35.879928 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.879906    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:04:35.885370 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.885349    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 00:04:35.886398 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.886380    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:04:35.887013 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.886997    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:04:35.887125 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.887106    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:35.887931 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.887892    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:04:35.887931 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.887930    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:04:35.888066 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.887958    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:04:35.888066 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.887986    2579 factory.go:55] Registering systemd factory
Nov 03 00:04:35.888066 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888002    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:04:35.888250 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888235    2579 factory.go:153] Registering CRI-O factory
Nov 03 00:04:35.888250 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888250    2579 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:04:35.888364 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888312    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:04:35.888364 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888338    2579 factory.go:103] Registering Raw factory
Nov 03 00:04:35.888364 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888344    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:04:35.888364 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888354    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:04:35.888364 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.888364    2579 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:04:35.889458 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.889444    2579 manager.go:319] Starting recovery of all containers
Nov 03 00:04:35.900979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.900957    2579 manager.go:324] Recovery completed
Nov 03 00:04:35.902693 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.902663    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:04:35.904243 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.904211    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 00:04:35.908123 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.908109    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:35.909372 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.909331    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 00:04:35.909699 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.909678    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zzt8c"
Nov 03 00:04:35.910096 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910082    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:35.910163 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910110    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:35.910163 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910120    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:35.910501 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.909360    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-253.ec2.internal.18e46027d9fc7616  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-253.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-253.ec2.internal,},FirstTimestamp:2026-11-03 00:04:35.877492246 +0000 UTC m=+0.402207902,LastTimestamp:2026-11-03 00:04:35.877492246 +0000 UTC m=+0.402207902,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-253.ec2.internal,}"
Nov 03 00:04:35.910719 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910699    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:04:35.910719 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910716    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:35.910827 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910716    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:04:35.910827 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.910775    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:04:35.911854 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.911838    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:35.911932 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.911869    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:35.911932 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.911883    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:35.913070 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.913056    2579 policy_none.go:50] "Start"
Nov 03 00:04:35.913108 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.913078    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:04:35.913108 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.913089    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:04:35.914973 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.914960    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zzt8c"
Nov 03 00:04:35.916493 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.916477    2579 policy_none.go:44] "Start"
Nov 03 00:04:35.930745 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.930722    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 00:04:35.930745 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.930741    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 00:04:35.930953 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.930931    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 00:04:35.931028 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.931012    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 00:04:35.944619 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.944546    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-253.ec2.internal.18e46027dbedf6ab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-253.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-253.ec2.internal,},FirstTimestamp:2026-11-03 00:04:35.910096555 +0000 UTC m=+0.434812210,LastTimestamp:2026-11-03 00:04:35.910096555 +0000 UTC m=+0.434812210,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-253.ec2.internal,}"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.961545    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.963932    2579 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.964017    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.964028    2579 server.go:87] "Starting device plugin registration server"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.964287    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.964299    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.964386    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.964488    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.964498    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.965068    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.965110    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.970267    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.971382    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.971398    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.971413    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:04:35.973234 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:35.971439    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 00:04:35.985582 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:35.985566    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:36.064505 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.064453    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:36.065354 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.065331    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:36.065417 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.065362    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:36.065417 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.065375    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:36.065417 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.065403    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.071553 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.071533    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"]
Nov 03 00:04:36.071603 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.071598    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:36.072349 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.072334    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:36.072407 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.072361    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:36.072407 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.072370    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:36.074570 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.074558    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:36.074724 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.074710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.074774 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.074738    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:36.075321 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.075304    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:36.075425 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.075327    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:36.075425 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.075336    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:36.075425 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.075368    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:36.075425 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.075394    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:36.075425 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.075407    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:36.077486 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.077469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.077558 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.077504    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:36.078182 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.078157    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:36.078272 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.078187    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:36.078272 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.078202    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:36.083763 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.083748    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.083833 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:36.083768    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-253.ec2.internal\": node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:36.106554 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:36.106392    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-253.ec2.internal\" not found" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.109059 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:36.109042    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-253.ec2.internal\" not found" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.125432 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:36.125411    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:36.190088 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.190058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.190088 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.190088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.190263 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.190115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e107905b7a9295ebc233864ba55f6a44-config\") pod \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" (UID: \"e107905b7a9295ebc233864ba55f6a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.226431 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:36.226409    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:36.290628 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.290603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.290746 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.290633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.290746 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.290663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e107905b7a9295ebc233864ba55f6a44-config\") pod \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" (UID: \"e107905b7a9295ebc233864ba55f6a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.290746 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.290725    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e107905b7a9295ebc233864ba55f6a44-config\") pod \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" (UID: \"e107905b7a9295ebc233864ba55f6a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.290746 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.290736    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.290908 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:36.290793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:36.327107 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:36.327052    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:37.622071 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:37.622051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:37.624211 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:37.624192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:37.636778 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:37.636757    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:37.737273 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:37.737235    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:37.837813 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:37.837743    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:37.938239 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:37.938202    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:38.029758 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.029733    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 00:04:38.030161 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.029872    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 00:04:38.038413 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:38.038393    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:38.130294 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.130214    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:37 +0000 UTC" deadline="2028-05-06 19:43:41.415096877 +0000 UTC"
Nov 03 00:04:38.130294 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.130258    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13219h39m3.284842369s"
Nov 03 00:04:38.139346 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:38.139318    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:04:38.149130 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.149104    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:38.162508 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.162491    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:38.176522 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:38.176479    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e361fc4461f10d60238f71bbc39227.slice/crio-43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7 WatchSource:0}: Error finding container 43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7: Status 404 returned error can't find the container with id 43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7
Nov 03 00:04:38.177113 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:38.177093    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode107905b7a9295ebc233864ba55f6a44.slice/crio-ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826 WatchSource:0}: Error finding container ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826: Status 404 returned error can't find the container with id ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826
Nov 03 00:04:38.185488 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.184300    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:04:38.186325 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.186280    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal" event={"ID":"e107905b7a9295ebc233864ba55f6a44","Type":"ContainerStarted","Data":"ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826"}
Nov 03 00:04:38.187174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.187155    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" event={"ID":"09e361fc4461f10d60238f71bbc39227","Type":"ContainerStarted","Data":"43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7"}
Nov 03 00:04:38.197495 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.197475    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:38.219292 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.219272    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:04:38.220355 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.220344    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:04:38.240383 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.240366    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:04:38.365746 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:38.365713    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:39.068934 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.068891    2579 apiserver.go:52] "Watching apiserver"
Nov 03 00:04:39.096737 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.096702    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 00:04:39.109461 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.109415    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:04:39.109897 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.109866    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq","openshift-multus/multus-7dkm6","kube-system/konnectivity-agent-4zf7k","openshift-dns/node-resolver-5jkfg","openshift-network-diagnostics/network-check-target-vlgrc","openshift-ovn-kubernetes/ovnkube-node-tnptp","kube-system/global-pull-secret-syncer-9rxv7","openshift-cluster-node-tuning-operator/tuned-tddjp","openshift-image-registry/node-ca-96xx2","openshift-multus/multus-additional-cni-plugins-v7x7d","openshift-multus/network-metrics-daemon-t4t4w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal","openshift-network-operator/iptables-alerter-s2gsr"]
Nov 03 00:04:39.112911 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.112887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.115214 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.115193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.117727 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.117707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.119837 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.119819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.123060 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.122278    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.124492 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.124472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:39.124596 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.124542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:39.127155 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.127136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.128170 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.128110    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:04:39.128654 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.128635    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:04:39.128754 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.128703    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.128754 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.128728    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gt4zj\""
Nov 03 00:04:39.128872 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.128790    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:04:39.128872 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.128794    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.129318 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.129301    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.129445 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.129416    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:04:39.129604 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.129585    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xfsv5\""
Nov 03 00:04:39.129816 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.129795    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:04:39.129895 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.129875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.131715 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.131696    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.134345 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.134327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.136340 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.136319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.136496 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.136393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:39.136908 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.136755    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nrnbx\""
Nov 03 00:04:39.139567 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.139549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.139661 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.139617    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:39.140160 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.140137    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pr6ld\""
Nov 03 00:04:39.140237 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.140169    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:04:39.140237 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.140233    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:04:39.140319 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.140298    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:04:39.140350 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.140314    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:04:39.140962 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.140941    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.141196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.141179    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.141265 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.141245    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzkq\""
Nov 03 00:04:39.141363 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.141345    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:04:39.141643 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.141630    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.144516 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.144496    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:04:39.144617 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.144566    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx2pd"
Nov 03 00:04:39.151357 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.151336    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sx2pd"
Nov 03 00:04:39.199252 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.199224    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:04:39.209014 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.208986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-netd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-env-overrides\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-cnibin\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.209172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.209172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-netns\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.209172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-kubelet\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209390 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-systemd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209390 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-var-lib-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209390 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-script-lib\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209390 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.209390 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-registration-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.209390 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-device-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-tmp\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-systemd-units\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-bin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-multus\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtx5\" (UniqueName: \"kubernetes.io/projected/a91bc30d-2c15-48ab-97c9-c54713750b36-kube-api-access-fxtx5\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frwz\" (UniqueName: \"kubernetes.io/projected/5c2280c9-7198-40c7-b6ac-fd299946a0d5-kube-api-access-7frwz\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.209640 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-kubelet\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-conf-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209741    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-kubelet-config\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mksl\" (UniqueName: \"kubernetes.io/projected/887b3093-6795-4b74-aa85-f46f3dbc76fd-kube-api-access-9mksl\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-run\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2hjp\" (UniqueName: \"kubernetes.io/projected/92908e9f-c806-49f8-8545-b4f2ff364cc4-kube-api-access-v2hjp\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.209893 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-k8s-cni-cncf-io\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a91bc30d-2c15-48ab-97c9-c54713750b36-hosts-file\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-slash\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-log-socket\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.209971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-agent-certs\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysconfig\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-lib-modules\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210130    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gsk\" (UniqueName: \"kubernetes.io/projected/f9c9783e-067c-456e-b511-e687ca741b33-kube-api-access-f4gsk\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210192 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-node-log\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8q6t\" (UniqueName: \"kubernetes.io/projected/e1479ca0-d25e-42a0-96a4-877e39edcdbb-kube-api-access-l8q6t\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-cni-binary-copy\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-tuned\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-bin\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-modprobe-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210387    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2kg\" (UniqueName: \"kubernetes.io/projected/4b2e7c3b-981c-436b-a1cb-563254e2b11f-kube-api-access-cf2kg\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-ovn\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-config\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210519 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-multus-certs\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-etc-kubernetes\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-socket-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-systemd\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b2e7c3b-981c-436b-a1cb-563254e2b11f-iptables-alerter-script\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2e7c3b-981c-436b-a1cb-563254e2b11f-host-slash\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovn-node-metrics-cert\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hnz4\" (UniqueName: \"kubernetes.io/projected/0d387007-75e5-4186-8039-b96c77d34f1f-kube-api-access-4hnz4\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-netns\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.210910 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-hostroot\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-dbus\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.210995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-konnectivity-ca\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91bc30d-2c15-48ab-97c9-c54713750b36-tmp-dir\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-sys\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-socket-dir-parent\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-etc-selinux\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-sys-fs\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.211239 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/887b3093-6795-4b74-aa85-f46f3dbc76fd-serviceca\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-etc-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-multus-daemon-config\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5vhh\" (UniqueName: \"kubernetes.io/projected/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kube-api-access-c5vhh\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-kubernetes\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-system-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-cnibin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-conf\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-var-lib-kubelet\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-host\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.211631 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.211587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/887b3093-6795-4b74-aa85-f46f3dbc76fd-host\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.229379 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229352    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:04:39.229501 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229365    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.229541 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229509    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.229692 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229610    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.229692 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229665    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.229902 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229730    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:04:39.229902 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229745    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.229902 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229753    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.229902 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229781    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9mnsd\""
Nov 03 00:04:39.229902 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229801    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qr8z8\""
Nov 03 00:04:39.230106 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229922    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:04:39.230106 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229949    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:04:39.230106 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.229960    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:04:39.230106 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.230017    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jtdbf\""
Nov 03 00:04:39.230106 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.230086    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqxtt\""
Nov 03 00:04:39.312193 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-var-lib-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-script-lib\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-registration-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-device-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-tmp\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-systemd-units\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-bin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-multus\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.312388 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtx5\" (UniqueName: \"kubernetes.io/projected/a91bc30d-2c15-48ab-97c9-c54713750b36-kube-api-access-fxtx5\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frwz\" (UniqueName: \"kubernetes.io/projected/5c2280c9-7198-40c7-b6ac-fd299946a0d5-kube-api-access-7frwz\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-kubelet\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-conf-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-kubelet-config\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mksl\" (UniqueName: \"kubernetes.io/projected/887b3093-6795-4b74-aa85-f46f3dbc76fd-kube-api-access-9mksl\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-run\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2hjp\" (UniqueName: \"kubernetes.io/projected/92908e9f-c806-49f8-8545-b4f2ff364cc4-kube-api-access-v2hjp\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-k8s-cni-cncf-io\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a91bc30d-2c15-48ab-97c9-c54713750b36-hosts-file\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-slash\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-log-socket\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-agent-certs\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysconfig\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-lib-modules\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.312850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gsk\" (UniqueName: \"kubernetes.io/projected/f9c9783e-067c-456e-b511-e687ca741b33-kube-api-access-f4gsk\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-node-log\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8q6t\" (UniqueName: \"kubernetes.io/projected/e1479ca0-d25e-42a0-96a4-877e39edcdbb-kube-api-access-l8q6t\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-cni-binary-copy\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-tuned\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.312998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-bin\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-modprobe-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2kg\" (UniqueName: \"kubernetes.io/projected/4b2e7c3b-981c-436b-a1cb-563254e2b11f-kube-api-access-cf2kg\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-ovn\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-config\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-multus-certs\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-etc-kubernetes\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-socket-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-systemd\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.313647 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b2e7c3b-981c-436b-a1cb-563254e2b11f-iptables-alerter-script\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2e7c3b-981c-436b-a1cb-563254e2b11f-host-slash\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovn-node-metrics-cert\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnz4\" (UniqueName: \"kubernetes.io/projected/0d387007-75e5-4186-8039-b96c77d34f1f-kube-api-access-4hnz4\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-netns\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-hostroot\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-dbus\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-konnectivity-ca\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91bc30d-2c15-48ab-97c9-c54713750b36-tmp-dir\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-sys\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-socket-dir-parent\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-etc-selinux\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-sys-fs\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.314400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/887b3093-6795-4b74-aa85-f46f3dbc76fd-serviceca\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-etc-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-multus-daemon-config\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5vhh\" (UniqueName: \"kubernetes.io/projected/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kube-api-access-c5vhh\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-kubernetes\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-system-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-cnibin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.313979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-conf\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-var-lib-kubelet\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-host\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/887b3093-6795-4b74-aa85-f46f3dbc76fd-host\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-netd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-env-overrides\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-cnibin\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.315171 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-netns\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-kubelet\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-systemd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314365    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-systemd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.314417    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-var-lib-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-etc-kubernetes\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315301    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-slash\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315353    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-kubelet\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315447    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-kubelet-config\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315534    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-run\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315688    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-conf\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-bin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315762    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-multus\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-script-lib\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-var-lib-kubelet\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316081    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-host\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.316174 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316101    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-conf-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316134    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-netd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315751    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-etc-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316244    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-cnibin\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316244    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-socket-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316248    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.315805    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-registration-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316292    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-systemd\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316309    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-device-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316349    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2e7c3b-981c-436b-a1cb-563254e2b11f-host-slash\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316405    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-netns\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316501    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-kubelet\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316534    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316669    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316741    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-env-overrides\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/887b3093-6795-4b74-aa85-f46f3dbc76fd-host\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316828    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-cni-binary-copy\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.316979 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/887b3093-6795-4b74-aa85-f46f3dbc76fd-serviceca\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.316934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317228    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-k8s-cni-cncf-io\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a91bc30d-2c15-48ab-97c9-c54713750b36-hosts-file\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317353    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-netns\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317417    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317490    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-socket-dir-parent\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317547    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-sys\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317602    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-lib-modules\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317674    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-hostroot\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317691    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-log-socket\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317731    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b2e7c3b-981c-436b-a1cb-563254e2b11f-iptables-alerter-script\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-kubernetes\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.317877 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317857    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-dbus\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.317893    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-system-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318050    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-multus-daemon-config\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318226    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318285    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91bc30d-2c15-48ab-97c9-c54713750b36-tmp-dir\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-cnibin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-konnectivity-ca\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.318449    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-bin\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318509    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.318530    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret podName:6a523c36-7d66-4549-b5fd-a9bc4903f53f nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.818511065 +0000 UTC m=+3.133711437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret") pod "global-pull-secret-syncer-9rxv7" (UID: "6a523c36-7d66-4549-b5fd-a9bc4903f53f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:39.318598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318533    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-etc-selinux\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318600    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-ovn\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318619    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-modprobe-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318719    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318723    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-systemd-units\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.318760    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysconfig\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.319007    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-node-log\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.319043    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-sys-fs\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.319094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-multus-certs\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.319127    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:39.319196 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.319142    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-config\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.319701 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.319253    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.819236332 +0000 UTC m=+3.134436698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:39.319701 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.319656    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.322495 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.322474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-agent-certs\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.322985 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.322961    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-tmp\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.323081 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.323001    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-tuned\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.323185 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.323166    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovn-node-metrics-cert\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.338422 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.337624    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtx5\" (UniqueName: \"kubernetes.io/projected/a91bc30d-2c15-48ab-97c9-c54713750b36-kube-api-access-fxtx5\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.338422 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.338173    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mksl\" (UniqueName: \"kubernetes.io/projected/887b3093-6795-4b74-aa85-f46f3dbc76fd-kube-api-access-9mksl\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.338422 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.338332    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:39.338422 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.338350    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:39.338422 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.338366    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p49qh for pod openshift-network-diagnostics/network-check-target-vlgrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:39.338765 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.338452    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh podName:a0405b2f-5102-4390-a855-a14f0cad58e5 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:39.838416758 +0000 UTC m=+3.153617132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p49qh" (UniqueName: "kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh") pod "network-check-target-vlgrc" (UID: "a0405b2f-5102-4390-a855-a14f0cad58e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:39.339121 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.339090    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2hjp\" (UniqueName: \"kubernetes.io/projected/92908e9f-c806-49f8-8545-b4f2ff364cc4-kube-api-access-v2hjp\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.341947 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.341902    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8q6t\" (UniqueName: \"kubernetes.io/projected/e1479ca0-d25e-42a0-96a4-877e39edcdbb-kube-api-access-l8q6t\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.342556 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.342534    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5vhh\" (UniqueName: \"kubernetes.io/projected/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kube-api-access-c5vhh\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.342765 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.342722    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2kg\" (UniqueName: \"kubernetes.io/projected/4b2e7c3b-981c-436b-a1cb-563254e2b11f-kube-api-access-cf2kg\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.354111 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.354039    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frwz\" (UniqueName: \"kubernetes.io/projected/5c2280c9-7198-40c7-b6ac-fd299946a0d5-kube-api-access-7frwz\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.354226 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.354102    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnz4\" (UniqueName: \"kubernetes.io/projected/0d387007-75e5-4186-8039-b96c77d34f1f-kube-api-access-4hnz4\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.358996 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.358971    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gsk\" (UniqueName: \"kubernetes.io/projected/f9c9783e-067c-456e-b511-e687ca741b33-kube-api-access-f4gsk\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.452012 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.451983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7dkm6"
Nov 03 00:04:39.457332 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.457307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:39.461997 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.461978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:39.466454 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.466419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:04:39.470320 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.470302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:04:39.473195 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.473146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:04:39.476220 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.476190    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc07a6d_27f1_4b19_85e0_bbb1012fa1b3.slice/crio-f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7 WatchSource:0}: Error finding container f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7: Status 404 returned error can't find the container with id f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7
Nov 03 00:04:39.477074 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.477043    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d387007_75e5_4186_8039_b96c77d34f1f.slice/crio-bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0 WatchSource:0}: Error finding container bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0: Status 404 returned error can't find the container with id bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0
Nov 03 00:04:39.477548 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.477526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:04:39.488033 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.488014    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92908e9f_c806_49f8_8545_b4f2ff364cc4.slice/crio-853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7 WatchSource:0}: Error finding container 853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7: Status 404 returned error can't find the container with id 853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7
Nov 03 00:04:39.490105 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.489177    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:04:39.491160 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.491142    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2280c9_7198_40c7_b6ac_fd299946a0d5.slice/crio-081d13c4d9ebf14ab7fe9d67b5e69e9cdadd9291e1b71533c03af417ddec0e8b WatchSource:0}: Error finding container 081d13c4d9ebf14ab7fe9d67b5e69e9cdadd9291e1b71533c03af417ddec0e8b: Status 404 returned error can't find the container with id 081d13c4d9ebf14ab7fe9d67b5e69e9cdadd9291e1b71533c03af417ddec0e8b
Nov 03 00:04:39.491284 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.491269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:04:39.493016 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.492523    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2e7c3b_981c_436b_a1cb_563254e2b11f.slice/crio-cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c WatchSource:0}: Error finding container cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c: Status 404 returned error can't find the container with id cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c
Nov 03 00:04:39.504173 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.503945    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4f37ba_6131_49ba_bd2a_7ecf1dded60e.slice/crio-f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f WatchSource:0}: Error finding container f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f: Status 404 returned error can't find the container with id f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f
Nov 03 00:04:39.507577 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:04:39.507552    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887b3093_6795_4b74_aa85_f46f3dbc76fd.slice/crio-cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d WatchSource:0}: Error finding container cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d: Status 404 returned error can't find the container with id cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d
Nov 03 00:04:39.612362 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.612292    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.918869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.918921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:39.918950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919112    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919134    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919147    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p49qh for pod openshift-network-diagnostics/network-check-target-vlgrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh podName:a0405b2f-5102-4390-a855-a14f0cad58e5 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.91918781 +0000 UTC m=+4.234388168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p49qh" (UniqueName: "kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh") pod "network-check-target-vlgrc" (UID: "a0405b2f-5102-4390-a855-a14f0cad58e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919687    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919740    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret podName:6a523c36-7d66-4549-b5fd-a9bc4903f53f nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.919725475 +0000 UTC m=+4.234925830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret") pod "global-pull-secret-syncer-9rxv7" (UID: "6a523c36-7d66-4549-b5fd-a9bc4903f53f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919799    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:39.919905 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:39.919829    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:40.91981846 +0000 UTC m=+4.235018829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:40.155063 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.154946    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-06-07 06:53:43.412525702 +0000 UTC"
Nov 03 00:04:40.155063 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.155006    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13974h49m3.257523406s"
Nov 03 00:04:40.198480 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.198155    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal" event={"ID":"e107905b7a9295ebc233864ba55f6a44","Type":"ContainerStarted","Data":"f1f455723cee090719c00f297b2b9476407415ebfe916a190f489976c2651fbe"}
Nov 03 00:04:40.232287 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.232252    2579 generic.go:356] "Generic (PLEG): container finished" podID="09e361fc4461f10d60238f71bbc39227" containerID="8a2f9beb06675417af14d4bffc20de8b6611af485012957088df0a440ab50f7f" exitCode=0
Nov 03 00:04:40.232493 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.232364    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" event={"ID":"09e361fc4461f10d60238f71bbc39227","Type":"ContainerDied","Data":"8a2f9beb06675417af14d4bffc20de8b6611af485012957088df0a440ab50f7f"}
Nov 03 00:04:40.275011 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.274952    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal" podStartSLOduration=2.274933935 podStartE2EDuration="2.274933935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:40.236640599 +0000 UTC m=+3.551840972" watchObservedRunningTime="2026-11-03 00:04:40.274933935 +0000 UTC m=+3.590134307"
Nov 03 00:04:40.291806 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.291672    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-96xx2" event={"ID":"887b3093-6795-4b74-aa85-f46f3dbc76fd","Type":"ContainerStarted","Data":"cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d"}
Nov 03 00:04:40.308872 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.308838    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tddjp" event={"ID":"5c2280c9-7198-40c7-b6ac-fd299946a0d5","Type":"ContainerStarted","Data":"081d13c4d9ebf14ab7fe9d67b5e69e9cdadd9291e1b71533c03af417ddec0e8b"}
Nov 03 00:04:40.321457 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.321352    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerStarted","Data":"853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7"}
Nov 03 00:04:40.329575 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.329491    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7dkm6" event={"ID":"0d387007-75e5-4186-8039-b96c77d34f1f","Type":"ContainerStarted","Data":"bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0"}
Nov 03 00:04:40.332076 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.331996    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zf7k" event={"ID":"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3","Type":"ContainerStarted","Data":"f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7"}
Nov 03 00:04:40.334655 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.334590    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq" event={"ID":"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e","Type":"ContainerStarted","Data":"f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f"}
Nov 03 00:04:40.348886 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.348859    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2gsr" event={"ID":"4b2e7c3b-981c-436b-a1cb-563254e2b11f","Type":"ContainerStarted","Data":"cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c"}
Nov 03 00:04:40.364177 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.364103    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jkfg" event={"ID":"a91bc30d-2c15-48ab-97c9-c54713750b36","Type":"ContainerStarted","Data":"3d97ffe5a33c1192bdd09894beb44be107315be716bb756de0d0f596aa5bf919"}
Nov 03 00:04:40.382913 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.382880    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"89cef0df2aef88f17850600d694119afb4f0ad0e63afa81e6dad72c1f2df76ee"}
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.927556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.927603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:40.927633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.927764    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.927786    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.927798    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p49qh for pod openshift-network-diagnostics/network-check-target-vlgrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.927862    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh podName:a0405b2f-5102-4390-a855-a14f0cad58e5 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:42.927844319 +0000 UTC m=+6.243044695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p49qh" (UniqueName: "kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh") pod "network-check-target-vlgrc" (UID: "a0405b2f-5102-4390-a855-a14f0cad58e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.928244    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.928301    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret podName:6a523c36-7d66-4549-b5fd-a9bc4903f53f nodeName:}" failed. No retries permitted until 2026-11-03 00:04:42.928284278 +0000 UTC m=+6.243484636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret") pod "global-pull-secret-syncer-9rxv7" (UID: "6a523c36-7d66-4549-b5fd-a9bc4903f53f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.928357    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:40.928484 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:40.928392    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:42.928380231 +0000 UTC m=+6.243580598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:41.156132 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.156036    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-06-25 02:00:29.99241825 +0000 UTC"
Nov 03 00:04:41.156132 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.156095    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h55m48.836327962s"
Nov 03 00:04:41.183148 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.182544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:41.183148 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:41.182651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:41.183148 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.182976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:41.183148 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:41.183059    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:41.183581 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.183475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:41.183581 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:41.183544    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:41.402303 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.402249    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" event={"ID":"09e361fc4461f10d60238f71bbc39227","Type":"ContainerStarted","Data":"2e268be7f92dfea96d9aefc9ab7d29e5d3dedc76f4f633866f288d975c336850"}
Nov 03 00:04:41.440698 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:41.440597    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" podStartSLOduration=3.440578142 podStartE2EDuration="3.440578142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:41.439559924 +0000 UTC m=+4.754760299" watchObservedRunningTime="2026-11-03 00:04:41.440578142 +0000 UTC m=+4.755778519"
Nov 03 00:04:42.410758 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:42.410661    2579 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="06705207f4a19fe3f293ff72786e66f8dce58a52fad94b2c024923dbb9f9c98a" exitCode=0
Nov 03 00:04:42.411307 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:42.411257    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerDied","Data":"06705207f4a19fe3f293ff72786e66f8dce58a52fad94b2c024923dbb9f9c98a"}
Nov 03 00:04:42.945272 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:42.945235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:42.945464 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:42.945282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:42.945464 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:42.945313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:42.945570 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.945472    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:42.945570 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.945492    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:42.945570 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.945505    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p49qh for pod openshift-network-diagnostics/network-check-target-vlgrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:42.945570 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.945555    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh podName:a0405b2f-5102-4390-a855-a14f0cad58e5 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:46.945538881 +0000 UTC m=+10.260739247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p49qh" (UniqueName: "kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh") pod "network-check-target-vlgrc" (UID: "a0405b2f-5102-4390-a855-a14f0cad58e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:42.945953 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.945934    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:42.946049 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.945987    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret podName:6a523c36-7d66-4549-b5fd-a9bc4903f53f nodeName:}" failed. No retries permitted until 2026-11-03 00:04:46.945972848 +0000 UTC m=+10.261173217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret") pod "global-pull-secret-syncer-9rxv7" (UID: "6a523c36-7d66-4549-b5fd-a9bc4903f53f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:42.946049 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.946046    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:42.946156 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:42.946080    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:46.94606944 +0000 UTC m=+10.261269822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:43.182088 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:43.181688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:43.182088 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:43.181809    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:43.182088 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:43.181865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:43.182088 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:43.181936    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:43.182088 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:43.181983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:43.182088 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:43.182057    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:43.422071 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:43.422027    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4zf7k" event={"ID":"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3","Type":"ContainerStarted","Data":"16e2a042cb7a8dd5ce92fa588df3e6de6417aa242197c96f5c00ec9697ce3dc7"}
Nov 03 00:04:44.291833 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:44.291688    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:44.292905 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:44.292229    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:04:44.328662 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:44.328588    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4zf7k" podStartSLOduration=3.628635589 podStartE2EDuration="7.328570735s" totalImagesPullingTime="3.699935146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:43.461157638 +0000 UTC m=+6.776358014" watchObservedRunningTime="2026-11-03 00:04:44.328570735 +0000 UTC m=+7.643771114"
Nov 03 00:04:45.187300 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:45.187269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:45.187867 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:45.187388    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:45.187867 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:45.187710    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:45.187867 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:45.187780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:45.187867 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:45.187819    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:45.187867 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:45.187864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:45.430186 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:45.429498    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-96xx2" event={"ID":"887b3093-6795-4b74-aa85-f46f3dbc76fd","Type":"ContainerStarted","Data":"b459fd4e3ba447542bea5b3062b3cdb64bb1e584d177fc44fc6259888022b347"}
Nov 03 00:04:45.432738 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:45.432255    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:04:45.432738 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:45.432663    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq" event={"ID":"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e","Type":"ContainerStarted","Data":"e74b38cf748a679df49d790019d34e0b2cec94b02b99bc1bf7fb2bb41d9cca05"}
Nov 03 00:04:46.542804 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:46.542761    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:46.984402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:46.984464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:46.984496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.984669    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.984687    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.984697    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p49qh for pod openshift-network-diagnostics/network-check-target-vlgrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.984740    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh podName:a0405b2f-5102-4390-a855-a14f0cad58e5 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.984726003 +0000 UTC m=+18.299926368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p49qh" (UniqueName: "kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh") pod "network-check-target-vlgrc" (UID: "a0405b2f-5102-4390-a855-a14f0cad58e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.985064    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:46.985223 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.985109    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret podName:6a523c36-7d66-4549-b5fd-a9bc4903f53f nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.985095315 +0000 UTC m=+18.300295671 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret") pod "global-pull-secret-syncer-9rxv7" (UID: "6a523c36-7d66-4549-b5fd-a9bc4903f53f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:46.988477 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.985952    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:46.988477 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:46.986043    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.986025088 +0000 UTC m=+18.301225463 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:47.179274 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.179158    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:04:46.542784336Z","UUID":"75b5757d-d3c9-4f55-91ca-c2efbf76b7d3","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:04:47.180804 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.180778    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:04:47.180933 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.180811    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:04:47.181089 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.181076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:47.181174 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:47.181158    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:47.181358 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.181344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:47.181547 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:47.181456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:47.181547 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.181339    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:47.181547 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:47.181542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:47.440034 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.439947    2579 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="d0ac352adbcedccf65b340e20bd89e7ca8dac7ff10e49bc180f92ec4a7f88e67" exitCode=0
Nov 03 00:04:47.440034 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.440023    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerDied","Data":"d0ac352adbcedccf65b340e20bd89e7ca8dac7ff10e49bc180f92ec4a7f88e67"}
Nov 03 00:04:47.443255 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.443224    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7dkm6" event={"ID":"0d387007-75e5-4186-8039-b96c77d34f1f","Type":"ContainerStarted","Data":"01b01d7f4f3ef100b9c314602812def3505ebc1cf39541c8ce0519a0b8bf88bb"}
Nov 03 00:04:47.445988 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.445960    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq" event={"ID":"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e","Type":"ContainerStarted","Data":"1ab145523fd23df78587c8c31fdeef199dc777762c6afe80021c2ee038231195"}
Nov 03 00:04:47.447891 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.447868    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jkfg" event={"ID":"a91bc30d-2c15-48ab-97c9-c54713750b36","Type":"ContainerStarted","Data":"005a71c914e5a129050b5ce10893a9f5ea331fc59ecd70fcd7c5869f2cd001e5"}
Nov 03 00:04:47.484109 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.484058    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-96xx2" podStartSLOduration=5.688427454 podStartE2EDuration="10.484040691s" totalImagesPullingTime="4.795613237s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:45.466398327 +0000 UTC m=+8.781598702" watchObservedRunningTime="2026-11-03 00:04:47.484040691 +0000 UTC m=+10.799241071"
Nov 03 00:04:47.526905 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.526850    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7dkm6" podStartSLOduration=3.154999452 podStartE2EDuration="10.526830966s" totalImagesPullingTime="7.371831514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.526449755 +0000 UTC m=+10.841650131" watchObservedRunningTime="2026-11-03 00:04:47.526830966 +0000 UTC m=+10.842031342"
Nov 03 00:04:47.563328 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:47.563270    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5jkfg" podStartSLOduration=2.839986279 podStartE2EDuration="10.563257068s" totalImagesPullingTime="7.723270789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:47.562699679 +0000 UTC m=+10.877900054" watchObservedRunningTime="2026-11-03 00:04:47.563257068 +0000 UTC m=+10.878457445"
Nov 03 00:04:48.452714 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:48.452473    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq" event={"ID":"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e","Type":"ContainerStarted","Data":"c1f11fb01b31e8ae692b48efd5ee2f3160f9739c6a73fbacd1ef2787af30c904"}
Nov 03 00:04:48.455855 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:48.455818    2579 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="9e1da9ef05a98cfdfa9008a992e094b5ea213f024d00ab5de93c32a93c9d6fc6" exitCode=0
Nov 03 00:04:48.456000 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:48.455935    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerDied","Data":"9e1da9ef05a98cfdfa9008a992e094b5ea213f024d00ab5de93c32a93c9d6fc6"}
Nov 03 00:04:48.491969 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:48.491920    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq" podStartSLOduration=3.809232293 podStartE2EDuration="11.491900273s" totalImagesPullingTime="7.68266798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:48.491264439 +0000 UTC m=+11.806464817" watchObservedRunningTime="2026-11-03 00:04:48.491900273 +0000 UTC m=+11.807100646"
Nov 03 00:04:49.185712 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:49.185173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:49.185712 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:49.185253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:49.185712 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:49.185274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:49.185712 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:49.185393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:49.185712 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:49.185537    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:49.185712 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:49.185634    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:49.460504 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:49.460406    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s2gsr" event={"ID":"4b2e7c3b-981c-436b-a1cb-563254e2b11f","Type":"ContainerStarted","Data":"535d9b8569edd3a07dcef146fb0fbef8df8509f436f9d0735b9c7f68590ef208"}
Nov 03 00:04:49.497551 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:49.497496    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s2gsr" podStartSLOduration=4.528114084 podStartE2EDuration="12.497476169s" totalImagesPullingTime="7.969362085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:49.496983746 +0000 UTC m=+12.812184122" watchObservedRunningTime="2026-11-03 00:04:49.497476169 +0000 UTC m=+12.812676542"
Nov 03 00:04:50.465214 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:50.465182    2579 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="545f8240e7d0111e7d9c8d349340afa128e806e40568165480f9080ee1630816" exitCode=0
Nov 03 00:04:50.465790 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:50.465277    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerDied","Data":"545f8240e7d0111e7d9c8d349340afa128e806e40568165480f9080ee1630816"}
Nov 03 00:04:50.467328 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:50.467293    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tddjp" event={"ID":"5c2280c9-7198-40c7-b6ac-fd299946a0d5","Type":"ContainerStarted","Data":"405833e8453b05c0d65961190c000bd47d79e54c1b2907bc7a7c37ab98c386aa"}
Nov 03 00:04:50.574850 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:50.574794    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tddjp" podStartSLOduration=3.405100799 podStartE2EDuration="13.574775682s" totalImagesPullingTime="10.169674883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:50.574295449 +0000 UTC m=+13.889495814" watchObservedRunningTime="2026-11-03 00:04:50.574775682 +0000 UTC m=+13.889976058"
Nov 03 00:04:51.183574 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:51.183540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:51.183766 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:51.183669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:51.183766 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:51.183736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:51.183868 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:51.183808    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:51.183868 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:51.183855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:51.183965 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:51.183930    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:52.473874 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:52.473835    2579 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="6a70306f11c060842e54c7b84c9bab9e46339a7453dbe1dad8e98c1145560d4d" exitCode=0
Nov 03 00:04:52.474634 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:52.473887    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerDied","Data":"6a70306f11c060842e54c7b84c9bab9e46339a7453dbe1dad8e98c1145560d4d"}
Nov 03 00:04:53.181600 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:53.181382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:53.181788 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:53.181404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:53.181788 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:53.181683    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:53.181788 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:53.181457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:53.181942 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:53.181788    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:53.181942 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:53.181904    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:53.478641 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:53.478537    2579 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="d450a71606b75862d0ab3aa73ef4a96084b2e977b6c1dd98510ec4e070cbdb79" exitCode=0
Nov 03 00:04:53.478641 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:53.478579    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerDied","Data":"d450a71606b75862d0ab3aa73ef4a96084b2e977b6c1dd98510ec4e070cbdb79"}
Nov 03 00:04:54.484083 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.483829    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" event={"ID":"92908e9f-c806-49f8-8545-b4f2ff364cc4","Type":"ContainerStarted","Data":"a6ba85596e3d27938b339420f4f14e4808571efafec73148af0379e3a6d1b9c8"}
Nov 03 00:04:54.486697 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.486676    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"0a51c3bc075b022ef4465cd1b36dbab826530f38d73ce231c6c7a06f4316e854"}
Nov 03 00:04:54.486771 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.486731    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"b3ab6000d05ea8607d78531cb5f27cfcc80cf2b4e932c8ddfbcf53f5a7b9ecb6"}
Nov 03 00:04:54.486771 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.486762    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"1e317682ef19545ce0539b10151bb89a52d0d716f5576c900fc419d21ea8cfcb"}
Nov 03 00:04:54.486835 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.486792    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"33a8a25633d8c1518d4606f01d33778512b2745f2d435ef72c31940348eea3c1"}
Nov 03 00:04:54.486835 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.486820    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"98d8c2549e12dc7a829c682180068099bee2c32d7f651e22b815a4045e7f8327"}
Nov 03 00:04:54.486889 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.486845    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"441441cf8868416cfc929bc14539892484e239e158f589f5ff997a8e0dd14b16"}
Nov 03 00:04:54.550172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:54.550127    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v7x7d" podStartSLOduration=7.306702085 podStartE2EDuration="17.550112719s" totalImagesPullingTime="10.243410634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:54.537830178 +0000 UTC m=+17.853030552" watchObservedRunningTime="2026-11-03 00:04:54.550112719 +0000 UTC m=+17.865313100"
Nov 03 00:04:55.047211 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:55.047182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:55.047363 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:55.047218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:55.047363 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:55.047240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:55.047363 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047329    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:55.047480 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047390    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret podName:6a523c36-7d66-4549-b5fd-a9bc4903f53f nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.047376811 +0000 UTC m=+34.362577168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret") pod "global-pull-secret-syncer-9rxv7" (UID: "6a523c36-7d66-4549-b5fd-a9bc4903f53f") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:55.047480 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047335    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:55.047480 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047470    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.047457663 +0000 UTC m=+34.362658032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:55.047480 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047337    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:55.047596 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047492    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:55.047596 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047501    2579 projected.go:196] Error preparing data for projected volume kube-api-access-p49qh for pod openshift-network-diagnostics/network-check-target-vlgrc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:55.047596 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.047531    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh podName:a0405b2f-5102-4390-a855-a14f0cad58e5 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.047524823 +0000 UTC m=+34.362725174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p49qh" (UniqueName: "kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh") pod "network-check-target-vlgrc" (UID: "a0405b2f-5102-4390-a855-a14f0cad58e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:55.184143 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:55.184117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:55.184283 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:55.184180    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:55.184283 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:55.184203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:55.184283 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.184273    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:55.184405 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.184309    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:55.184482 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:55.184413    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:56.494200 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:56.494115    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"0f64f62a67f05b9bb29bd228843fa1e2df6a3b69d136cf1c831ef31b8892b33b"}
Nov 03 00:04:57.182372 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:57.182338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:57.182545 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:57.182492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:57.182636 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:57.182536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:57.182692 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:57.182644    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:57.182692 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:57.182684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:57.182791 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:57.182768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:58.501525 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:58.501484    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" event={"ID":"e1479ca0-d25e-42a0-96a4-877e39edcdbb","Type":"ContainerStarted","Data":"dcb0963d98f47819a9aa17fd3e16740c50416089a39a3eb37742c799a8935e70"}
Nov 03 00:04:58.502277 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:58.501816    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:58.517191 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:58.517164    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:58.589619 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:58.589535    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" podStartSLOduration=7.214798596 podStartE2EDuration="21.589522701s" totalImagesPullingTime="14.374724105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:58.589242554 +0000 UTC m=+21.904442929" watchObservedRunningTime="2026-11-03 00:04:58.589522701 +0000 UTC m=+21.904723074"
Nov 03 00:04:59.181851 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:59.181814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:04:59.182028 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:59.181872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:04:59.182028 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:59.181875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:04:59.182028 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:59.181949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:04:59.182180 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:59.182047    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:04:59.182180 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:04:59.182159    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:04:59.503967 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:59.503892    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:04:59.504604 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:59.504394    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:04:59.525756 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:04:59.525726    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:00.419911 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.419593    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rxv7"]
Nov 03 00:05:00.420057 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.419953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:00.420134 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:00.420063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:05:00.422584 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.422558    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vlgrc"]
Nov 03 00:05:00.422669 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.422645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:00.422732 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:00.422716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:05:00.423033 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.423015    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4t4w"]
Nov 03 00:05:00.423107 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.423097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:00.423187 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:00.423171    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:05:00.509034 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:00.506966    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:01.509164 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:01.509136    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:02.181537 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:02.181512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:02.181730 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:02.181512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:02.181730 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:02.181623    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:05:02.181730 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:02.181512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:02.181829 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:02.181679    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:05:02.181829 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:02.181762    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:05:03.648485 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:03.648425    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:03.648860 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:03.648675    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:03.659535 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:03.659504    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" podUID="e1479ca0-d25e-42a0-96a4-877e39edcdbb" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 00:05:03.668038 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:03.668005    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp" podUID="e1479ca0-d25e-42a0-96a4-877e39edcdbb" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 00:05:04.181212 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:04.181174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:04.181372 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:04.181261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:04.181372 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:04.181269    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:05:04.181372 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:04.181351    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:05:04.181513 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:04.181394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:04.181513 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:04.181464    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:05:04.685623 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:04.685585    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:04.686077 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:04.685736    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:04.686281 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:04.686266    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:06.181575 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.181539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:06.181974 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.181548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:06.181974 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.181655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:06.181974 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:06.181661    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rxv7" podUID="6a523c36-7d66-4549-b5fd-a9bc4903f53f"
Nov 03 00:05:06.181974 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:06.181722    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vlgrc" podUID="a0405b2f-5102-4390-a855-a14f0cad58e5"
Nov 03 00:05:06.181974 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:06.181862    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4t4w" podUID="f9c9783e-067c-456e-b511-e687ca741b33"
Nov 03 00:05:06.443602 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.443531    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeReady"
Nov 03 00:05:06.443727 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.443653    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 00:05:06.502969 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.502932    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pzqlp"]
Nov 03 00:05:06.506049 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.506032    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sjlsx"]
Nov 03 00:05:06.506189 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.506169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.508379 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.508359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.524805 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.524783    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjlsx"]
Nov 03 00:05:06.525381 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.525363    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pzqlp"]
Nov 03 00:05:06.582021 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.581990    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:05:06.582021 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.582005    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:05:06.582021 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.582003    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:05:06.595154 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.595129    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:05:06.595311 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.595295    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ct6nh\""
Nov 03 00:05:06.595311 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.595305    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7b8cw\""
Nov 03 00:05:06.595393 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.595296    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:05:06.627606 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.627582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gwt\" (UniqueName: \"kubernetes.io/projected/24c625e0-13a7-48df-bd0e-c383ff56ecca-kube-api-access-98gwt\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.627713 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.627621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgzfs\" (UniqueName: \"kubernetes.io/projected/4d168306-103a-47b1-9b32-0c33b7825416-kube-api-access-dgzfs\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.627713 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.627657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.627781 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.627740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d168306-103a-47b1-9b32-0c33b7825416-cert\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.627781 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.627766    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c625e0-13a7-48df-bd0e-c383ff56ecca-tmp-dir\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.627899 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.627881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24c625e0-13a7-48df-bd0e-c383ff56ecca-config-volume\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.728931 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.728849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c625e0-13a7-48df-bd0e-c383ff56ecca-tmp-dir\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.728931 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.728929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24c625e0-13a7-48df-bd0e-c383ff56ecca-config-volume\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.729126 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.728962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gwt\" (UniqueName: \"kubernetes.io/projected/24c625e0-13a7-48df-bd0e-c383ff56ecca-kube-api-access-98gwt\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.729126 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.728991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzfs\" (UniqueName: \"kubernetes.io/projected/4d168306-103a-47b1-9b32-0c33b7825416-kube-api-access-dgzfs\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.729126 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.729021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.729126 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.729047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d168306-103a-47b1-9b32-0c33b7825416-cert\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.729314 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:06.729189    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:06.729314 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.729197    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c625e0-13a7-48df-bd0e-c383ff56ecca-tmp-dir\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.729314 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:06.729260    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.229232276 +0000 UTC m=+30.544432642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:06.729573 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.729553    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24c625e0-13a7-48df-bd0e-c383ff56ecca-config-volume\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.732832 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.732808    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d168306-103a-47b1-9b32-0c33b7825416-cert\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.750092 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.750066    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gwt\" (UniqueName: \"kubernetes.io/projected/24c625e0-13a7-48df-bd0e-c383ff56ecca-kube-api-access-98gwt\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:06.750172 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.750121    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzfs\" (UniqueName: \"kubernetes.io/projected/4d168306-103a-47b1-9b32-0c33b7825416-kube-api-access-dgzfs\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:06.825260 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:06.825230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:07.109742 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:07.109527    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:07.140732 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:07.140714    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjlsx"]
Nov 03 00:05:07.141819 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:05:07.141793    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d168306_103a_47b1_9b32_0c33b7825416.slice/crio-2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca WatchSource:0}: Error finding container 2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca: Status 404 returned error can't find the container with id 2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca
Nov 03 00:05:07.233185 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:07.233157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:07.233517 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:07.233257    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:07.233517 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:07.233307    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.233294589 +0000 UTC m=+31.548494941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:07.269217 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:07.269186    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:07.523144 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:07.523067    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjlsx" event={"ID":"4d168306-103a-47b1-9b32-0c33b7825416","Type":"ContainerStarted","Data":"2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca"}
Nov 03 00:05:08.181874 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.181840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:08.182039 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.181884    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:08.182092 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.182076    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:08.196920 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.196899    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:05:08.197015 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.196988    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:05:08.209342 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.209321    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:05:08.209501 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.209351    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:05:08.209501 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.209393    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:05:08.209501 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.209412    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:05:08.209501 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.209421    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:05:08.241819 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:08.241799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:08.242158 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:08.241909    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:08.242158 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:08.241955    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:10.241940925 +0000 UTC m=+33.557141276 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:10.256140 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:10.256119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:10.256478 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:10.256300    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:10.256478 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:10.256371    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.256347047 +0000 UTC m=+37.571547418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:10.288331 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:10.288299    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:10.384584 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:10.384517    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:10.531445 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:10.531401    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjlsx" event={"ID":"4d168306-103a-47b1-9b32-0c33b7825416","Type":"ContainerStarted","Data":"ed21facd85ca78d7378b2b8c146f7b558ac9153d8282cf26d3da41447cf366ca"}
Nov 03 00:05:10.569981 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:10.569932    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sjlsx" podStartSLOduration=1.555524223 podStartE2EDuration="4.569920416s" totalImagesPullingTime="3.014396193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:10.569328455 +0000 UTC m=+33.884528826" watchObservedRunningTime="2026-11-03 00:05:10.569920416 +0000 UTC m=+33.885120790"
Nov 03 00:05:11.061392 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.061353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:11.061392 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.061395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:11.061601 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.061414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:11.061601 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:11.061555    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:11.061660 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:11.061615    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:43.06159973 +0000 UTC m=+66.376800106 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : secret "metrics-daemon-secret" not found
Nov 03 00:05:11.064041 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.064015    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:11.064322 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.064299    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:11.206856 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.206825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:11.227124 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.227104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:11.555422 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:05:11.555388    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a523c36_7d66_4549_b5fd_a9bc4903f53f.slice/crio-b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e WatchSource:0}: Error finding container b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e: Status 404 returned error can't find the container with id b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e
Nov 03 00:05:11.555727 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.555499    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rxv7"]
Nov 03 00:05:11.582720 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:11.582700    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vlgrc"]
Nov 03 00:05:11.583075 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:05:11.583047    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0405b2f_5102_4390_a855_a14f0cad58e5.slice/crio-5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064 WatchSource:0}: Error finding container 5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064: Status 404 returned error can't find the container with id 5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064
Nov 03 00:05:12.537676 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.537638    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vlgrc" event={"ID":"a0405b2f-5102-4390-a855-a14f0cad58e5","Type":"ContainerStarted","Data":"5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064"}
Nov 03 00:05:12.538908 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.538875    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rxv7" event={"ID":"6a523c36-7d66-4549-b5fd-a9bc4903f53f","Type":"ContainerStarted","Data":"b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e"}
Nov 03 00:05:12.866826 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.866754    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"]
Nov 03 00:05:12.869673 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.869648    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:12.887055 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.887030    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4cnb6"]
Nov 03 00:05:12.887183 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.887170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:12.904564 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.904542    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 00:05:12.905793 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.905769    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"]
Nov 03 00:05:12.905915 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.905881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:12.911195 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.911157    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:12.911195 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.911167    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4v7ss\""
Nov 03 00:05:12.930968 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.930940    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-8whn6-nj4vj"]
Nov 03 00:05:12.945127 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.945103    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4cnb6"]
Nov 03 00:05:12.945339 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.945178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:12.961105 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.961073    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8whn6-nj4vj"]
Nov 03 00:05:12.975730 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.975703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:12.975856 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.975743    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5b7233c-5ec0-47fb-9129-793e2383b1bc-crio-socket\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:12.975916 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.975865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5b7233c-5ec0-47fb-9129-793e2383b1bc-data-volume\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:12.975916 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.975897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:12.976014 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.975941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6kh\" (UniqueName: \"kubernetes.io/projected/80f326a9-e6d5-45df-b03d-37d6bec25926-kube-api-access-5g6kh\") pod \"migrator-66df5d496b-dd54j\" (UID: \"80f326a9-e6d5-45df-b03d-37d6bec25926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:12.976087 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:12.976055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2sx\" (UniqueName: \"kubernetes.io/projected/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-api-access-mj2sx\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.004485 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004459    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7llq5\""
Nov 03 00:05:13.004598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004526    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:05:13.004598 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004597    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:05:13.004709 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004658    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 00:05:13.004755 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004711    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:05:13.004755 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004750    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:05:13.004954 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.004937    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rns5k\""
Nov 03 00:05:13.012165 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.012137    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:13.029578 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.029554    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 00:05:13.077145 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.077280 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.077280 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2sx\" (UniqueName: \"kubernetes.io/projected/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-api-access-mj2sx\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.077280 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.077280 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.077507 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5b7233c-5ec0-47fb-9129-793e2383b1bc-crio-socket\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.077507 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5b7233c-5ec0-47fb-9129-793e2383b1bc-data-volume\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.077507 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.077507 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6kh\" (UniqueName: \"kubernetes.io/projected/80f326a9-e6d5-45df-b03d-37d6bec25926-kube-api-access-5g6kh\") pod \"migrator-66df5d496b-dd54j\" (UID: \"80f326a9-e6d5-45df-b03d-37d6bec25926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:13.077507 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.077463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.077892 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.077876    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.077946 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.077936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.57791624 +0000 UTC m=+36.893116615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.078274 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.078236    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5b7233c-5ec0-47fb-9129-793e2383b1bc-crio-socket\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.078555 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.078538    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5b7233c-5ec0-47fb-9129-793e2383b1bc-data-volume\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.078643 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.078605    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.078700 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.078658    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:13.578641937 +0000 UTC m=+36.893842304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.098271 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.098239    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2sx\" (UniqueName: \"kubernetes.io/projected/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-api-access-mj2sx\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.101860 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.101834    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6kh\" (UniqueName: \"kubernetes.io/projected/80f326a9-e6d5-45df-b03d-37d6bec25926-kube-api-access-5g6kh\") pod \"migrator-66df5d496b-dd54j\" (UID: \"80f326a9-e6d5-45df-b03d-37d6bec25926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:13.110476 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.110449    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:13.178278 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.178216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.178421 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.178306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.178421 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.178337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.178421 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.178369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.184337 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.179993    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.184337 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.181801    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.184337 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.182018    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.197963 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.197938    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.213783 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.213590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:13.280635 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.280605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:13.536286 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.536258    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"]
Nov 03 00:05:13.537536 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:05:13.537510    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f326a9_e6d5_45df_b03d_37d6bec25926.slice/crio-a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39 WatchSource:0}: Error finding container a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39: Status 404 returned error can't find the container with id a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39
Nov 03 00:05:13.542838 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.542812    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j" event={"ID":"80f326a9-e6d5-45df-b03d-37d6bec25926","Type":"ContainerStarted","Data":"a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39"}
Nov 03 00:05:13.582777 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.582746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.582956 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.582798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:13.582956 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.582895    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.582956 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.582933    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.582956 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.582949    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.582934979 +0000 UTC m=+37.898135347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:13.583173 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:13.582993    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.582978059 +0000 UTC m=+37.898178430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:13.606759 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.606734    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-8whn6-nj4vj"]
Nov 03 00:05:13.608255 ip-10-0-133-253 kubenswrapper[2579]: W1103 00:05:13.608222    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821da046_5713_4974_ad87_8debd07cc8bf.slice/crio-df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65 WatchSource:0}: Error finding container df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65: Status 404 returned error can't find the container with id df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65
Nov 03 00:05:13.748661 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.748609    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:13.848041 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:13.847941    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:14.114799 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.114716    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-drrmw"]
Nov 03 00:05:14.133455 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.132872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.136037 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.135950    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-drrmw"]
Nov 03 00:05:14.148210 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.147790    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 00:05:14.148210 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.148015    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k88mc\""
Nov 03 00:05:14.148210 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.148103    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 00:05:14.148210 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.148139    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 00:05:14.148210 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.148188    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 00:05:14.287935 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.287894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-key\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.288128 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.287960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:14.288128 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.287990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-cabundle\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.288128 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.288096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btczm\" (UniqueName: \"kubernetes.io/projected/5f6b2892-309f-49f9-94e2-9af7a736d740-kube-api-access-btczm\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.288296 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:14.288266    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:14.288330 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:14.288320    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.28830488 +0000 UTC m=+45.603505250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:14.389219 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.389113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btczm\" (UniqueName: \"kubernetes.io/projected/5f6b2892-309f-49f9-94e2-9af7a736d740-kube-api-access-btczm\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.389219 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.389167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-key\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.389219 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.389208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-cabundle\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.390387 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.390359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-cabundle\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.392846 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.392807    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-key\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.417744 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.417677    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btczm\" (UniqueName: \"kubernetes.io/projected/5f6b2892-309f-49f9-94e2-9af7a736d740-kube-api-access-btczm\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.463707 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.463452    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:14.570386 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.570344    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8whn6-nj4vj" event={"ID":"821da046-5713-4974-ad87-8debd07cc8bf","Type":"ContainerStarted","Data":"df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65"}
Nov 03 00:05:14.583982 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.583952    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:14.591400 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.591369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:14.591552 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.591454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:14.591640 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:14.591614    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:14.591702 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:14.591685    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:16.59166606 +0000 UTC m=+39.906866412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:14.592116 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:14.592096    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:14.592231 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:14.592190    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:16.592134305 +0000 UTC m=+39.907334670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:14.689122 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.688995    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:14.793909 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:14.793882    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-drrmw"]
Nov 03 00:05:15.022772 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:15.022653    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:15.575633 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:15.575603    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j" event={"ID":"80f326a9-e6d5-45df-b03d-37d6bec25926","Type":"ContainerStarted","Data":"c7f6baad4ee4ca3fd67417896c0d107f8905fbb656e0cb67a27f63dde72429a5"}
Nov 03 00:05:15.577077 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:15.577046    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw" event={"ID":"5f6b2892-309f-49f9-94e2-9af7a736d740","Type":"ContainerStarted","Data":"bc6e2a29c3ab00719834c870960ea273378444c579e91a1e2b64e608531d77c8"}
Nov 03 00:05:15.615146 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:15.614509    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j" podStartSLOduration=1.9326971689999999 podStartE2EDuration="3.614489724s" totalImagesPullingTime="1.681792555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:15.614095821 +0000 UTC m=+38.929296196" watchObservedRunningTime="2026-11-03 00:05:15.614489724 +0000 UTC m=+38.929690095"
Nov 03 00:05:16.581059 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:16.580987    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vlgrc" event={"ID":"a0405b2f-5102-4390-a855-a14f0cad58e5","Type":"ContainerStarted","Data":"147d64e1836a1cabf4b06509a818c5e4fae71898e427fb7efd889d4275ec17de"}
Nov 03 00:05:16.608261 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:16.608219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:16.608463 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:16.608323    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:16.608463 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:16.608324    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:16.608463 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:16.608400    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.6083822 +0000 UTC m=+43.923582566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:16.608463 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:16.608416    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:16.608661 ip-10-0-133-253 kubenswrapper[2579]: E1103 00:05:16.608497    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.608478483 +0000 UTC m=+43.923678859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:16.619533 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:16.619453    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vlgrc" podStartSLOduration=35.638830143 podStartE2EDuration="39.619420624s" totalImagesPullingTime="3.980590481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:16.619186356 +0000 UTC m=+39.934386730" watchObservedRunningTime="2026-11-03 00:05:16.619420624 +0000 UTC m=+39.934621001"
Nov 03 00:05:16.782307 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:16.782272    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:16.881195 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:16.881126    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:17.586302 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:17.586222    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rxv7" event={"ID":"6a523c36-7d66-4549-b5fd-a9bc4903f53f","Type":"ContainerStarted","Data":"1b401e2c490ec5075ba4097386e82b4229d93e986eab2672c8d476e743080797"}
Nov 03 00:05:17.587815 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:17.587778    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8whn6-nj4vj" event={"ID":"821da046-5713-4974-ad87-8debd07cc8bf","Type":"ContainerStarted","Data":"b0a2825ffeb13cbf45a3335940e1fc10ef9663a0052950d3a3d476da1819d154"}
Nov 03 00:05:17.587944 ip-10-0-133-253 kubenswrapper[2579]: I1103 00:05:17.587928    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:17.611890 ip-10-0-133-253 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 00:05:17.630812 ip-10-0-133-253 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 00:05:17.630983 ip-10-0-133-253 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 00:05:17.631032 ip-10-0-133-253 systemd[1]: kubelet.service: Consumed 2.444s CPU time.
Nov 03 00:05:17.648175 ip-10-0-133-253 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:05:17.718028 ip-10-0-133-253 kubenswrapper[4550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:05:17.718028 ip-10-0-133-253 kubenswrapper[4550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:05:17.718028 ip-10-0-133-253 kubenswrapper[4550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:05:17.718028 ip-10-0-133-253 kubenswrapper[4550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:05:17.720152 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720065    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.720152 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720152    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.720152 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720156    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720159    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720163    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720166    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720169    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720180    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720183    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720186    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720189    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720192    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720194    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720198    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720200    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720203    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720206    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720208    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720211    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720214    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720217    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.720258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720219    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720224    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720228    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720231    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720234    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720237    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720240    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720243    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720245    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720249    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720252    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720255    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720257    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720260    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720263    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720266    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720269    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720272    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720275    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.720783 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720278    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720281    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720283    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720286    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720289    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720291    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720294    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720297    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720299    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720302    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720305    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720308    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720311    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720315    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720319    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720323    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720327    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720329    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720332    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.721278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720335    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720337    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720340    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720342    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720345    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720348    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720350    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720353    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720356    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720358    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720361    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720363    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720366    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720369    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720372    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720374    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720377    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720379    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720382    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720385    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.721765 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720387    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720390    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720392    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720395    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720398    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720401    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720403    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720406    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720408    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720411    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720414    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720416    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720419    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720421    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720424    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720439    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720442    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720445    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720447    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720450    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.722252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720453    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720456    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720459    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720461    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720465    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720468    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.720471    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721245    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721254    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721258    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721262    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721264    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721267    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721270    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721273    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721275    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721279    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721282    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721285    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.722771 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721287    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721290    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721294    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721297    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721299    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721302    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721305    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721307    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721310    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721313    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721315    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721318    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721321    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721324    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721326    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721329    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721331    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721334    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721336    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721339    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.723256 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721343    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721346    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721348    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721351    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721353    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721356    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721358    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721361    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721364    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721367    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721370    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721372    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721375    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721377    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721380    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721382    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721385    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721388    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721390    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721393    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.723768 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721395    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721398    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721400    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721403    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721405    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721408    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721410    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721413    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721415    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721418    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721421    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721423    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721426    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721444    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721449    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721452    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721454    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721457    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721460    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721463    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.724265 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721466    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721469    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721472    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721475    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721478    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721481    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721483    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721486    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721489    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721492    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721494    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721497    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721500    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721502    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721505    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721508    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721510    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721513    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721515    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.724782 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721518    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721521    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721524    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721527    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721529    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721532    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721534    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721557    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721560    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721563    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721567    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721571    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721574    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721577    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.721582    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722099    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722104    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722108    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722111    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.725246 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722114    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722117    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722119    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722122    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722125    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722127    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722130    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722133    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722135    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722138    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722142    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722146    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722149    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722152    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722154    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722157    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722159    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722162    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722165    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.725987 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722167    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722172    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722176    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722182    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722185    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722188    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722190    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722193    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722198    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722201    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722203    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722206    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722209    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722211    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722214    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722216    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722219    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722222    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722224    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.726536 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722227    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722230    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722232    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722235    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722237    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722240    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722252    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722256    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722259    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722262    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722265    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722268    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722270    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722273    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722276    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722280    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722283    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722286    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722288    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.727030 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722291    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722293    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722297    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722299    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722302    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722304    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722307    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722309    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722312    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722315    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722317    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722320    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722322    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722325    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722328    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722330    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722333    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722335    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722338    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722341    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.727513 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722343    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722346    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722348    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722353    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722355    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722358    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722361    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722363    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722365    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722369    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722372    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722374    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722377    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722380    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722383    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722386    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722389    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722392    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722394    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722397    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.728079 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722399    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722402    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722404    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722407    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.722409    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722515    4550 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722522    4550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722529    4550 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722535    4550 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722539    4550 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722543    4550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722548    4550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722552    4550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722555    4550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722558    4550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722562    4550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722565    4550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722569    4550 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722572    4550 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722575    4550 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722578    4550 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722581    4550 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722585    4550 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:05:17.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722590    4550 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722593    4550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722596    4550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722599    4550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722603    4550 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722610    4550 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722613    4550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722616    4550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722621    4550 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722624    4550 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722627    4550 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722630    4550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722633    4550 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722636    4550 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722641    4550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722644    4550 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722647    4550 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722650    4550 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722656    4550 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722659    4550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722664    4550 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722668    4550 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722671    4550 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722674    4550 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:05:17.729463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722677    4550 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722681    4550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722684    4550 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722687    4550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722691    4550 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722694    4550 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722697    4550 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722700    4550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722703    4550 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722707    4550 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722709    4550 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722712    4550 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722716    4550 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722719    4550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722723    4550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722726    4550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722729    4550 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722735    4550 flags.go:64] FLAG: --help="false"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722738    4550 flags.go:64] FLAG: --hostname-override="ip-10-0-133-253.ec2.internal"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722742    4550 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722746    4550 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722749    4550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722752    4550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:05:17.730519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722756    4550 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722759    4550 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722762    4550 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722767    4550 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722770    4550 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722773    4550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722776    4550 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722780    4550 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722783    4550 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722786    4550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722789    4550 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722792    4550 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722796    4550 flags.go:64] FLAG: --lock-file=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722799    4550 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722802    4550 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722805    4550 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722811    4550 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722814    4550 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722818    4550 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722821    4550 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722824    4550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722827    4550 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722830    4550 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722833    4550 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722838    4550 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:05:17.731215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722841    4550 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722846    4550 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722849    4550 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722852    4550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722855    4550 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722858    4550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722861    4550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722864    4550 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722867    4550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722874    4550 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722879    4550 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722882    4550 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722885    4550 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722888    4550 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722891    4550 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722894    4550 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722897    4550 flags.go:64] FLAG: --port="10250"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722900    4550 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722903    4550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b9fca6005c2918e4"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722906    4550 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722910    4550 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722913    4550 flags.go:64] FLAG: --register-node="true"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722916    4550 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722920    4550 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722923    4550 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722926    4550 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:05:17.731919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722929    4550 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722933    4550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722936    4550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722939    4550 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722942    4550 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722945    4550 flags.go:64] FLAG: --runonce="false"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722948    4550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722953    4550 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722956    4550 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722959    4550 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722962    4550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722965    4550 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722968    4550 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722971    4550 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722974    4550 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722977    4550 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722981    4550 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722984    4550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722988    4550 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722991    4550 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722993    4550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.722999    4550 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723002    4550 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723005    4550 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723009    4550 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:05:17.732597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723012    4550 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723014    4550 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723017    4550 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723020    4550 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723023    4550 flags.go:64] FLAG: --v="2"
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723027    4550 flags.go:64] FLAG: --version="false"
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723032    4550 flags.go:64] FLAG: --vmodule=""
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723035    4550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723039    4550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723190    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723194    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723198    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723201    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723204    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723208    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723210    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723213    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723216    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723219    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723221    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723224    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.733277 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723227    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723229    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723234    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723237    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723240    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723243    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723246    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723249    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723251    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723254    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723256    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723259    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723262    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723265    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723268    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723270    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723273    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723276    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723278    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.733825 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723281    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723285    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723288    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723290    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723293    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723296    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723299    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723302    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723305    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723307    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723310    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723313    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723315    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723318    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723321    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723325    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723327    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723330    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723333    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723336    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.734342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723338    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723341    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723355    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723359    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723363    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723366    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723369    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723372    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723375    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723378    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723380    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723382    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723385    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723388    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723391    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723394    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723397    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723401    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723406    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.735110 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723409    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723412    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723415    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723417    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723420    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723422    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723425    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723440    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723447    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723450    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723453    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723456    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723458    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723461    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723464    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723467    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723470    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723472    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723475    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.735676 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723477    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723480    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723483    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723485    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723488    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723490    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723493    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723496    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723498    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723501    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723504    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723507    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723511    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723514    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723516    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723519    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.736289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723522    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.723547    4550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723584    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723588    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723591    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723595    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723598    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723601    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723603    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723606    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723609    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723612    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723615    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723617    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723620    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723623    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.736781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723625    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723628    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723630    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723633    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723635    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723638    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723641    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723644    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723647    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723649    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723653    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723656    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723660    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723665    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723668    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723671    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723673    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723676    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723679    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.737177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723681    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723686    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723691    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723694    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723697    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723700    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723703    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723706    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723708    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723711    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723714    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723716    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723719    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723722    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723724    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723727    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723730    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723733    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723735    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.737796 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723738    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723741    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723743    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723746    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723749    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723752    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723755    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723758    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723761    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723764    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723766    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723769    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723772    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723775    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723778    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723782    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723785    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723787    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723790    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.738392 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723793    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723795    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723798    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723800    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723802    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723805    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723808    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723810    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723813    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723816    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723818    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723821    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723823    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723826    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723828    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723831    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723834    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723836    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723839    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.739132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723842    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723848    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723850    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723853    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723856    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723858    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723861    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723864    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723866    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723870    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723873    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723876    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723878    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723881    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723884    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.723887    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.728254    4550 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.728272    4550 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.728291    4550 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.728296    4550 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:05:17.739633 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728380    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728386    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728389    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728392    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728396    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728399    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728402    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728405    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728408    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728411    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728414    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728416    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728426    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728445    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728450    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728454    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728457    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728461    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728464    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.740123 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728467    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728469    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728472    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728475    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728477    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728480    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728483    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728486    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728489    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728491    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728494    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728497    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728501    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728505    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728509    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728511    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728515    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728517    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728520    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.740601 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728523    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728525    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728527    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728530    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728533    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728535    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728537    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728546    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728549    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728552    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728555    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728558    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728560    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728563    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728565    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728568    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728570    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728573    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728575    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.741111 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728578    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728581    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728585    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728588    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728591    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728594    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728597    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728599    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728602    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728604    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728607    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728609    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728612    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728615    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728618    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728621    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728624    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728627    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728629    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728632    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.741744 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728634    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728643    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728646    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728649    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728651    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728654    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728656    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728659    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728662    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728664    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728667    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728670    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728673    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728675    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728678    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728680    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728683    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728686    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728689    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728692    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.742422 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728695    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728697    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728700    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728702    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728705    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728707    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728710    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728712    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728715    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.728745    4550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728929    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728934    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728937    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728940    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728948    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728951    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.742979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728954    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728956    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728959    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728961    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728971    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728974    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728977    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728979    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728982    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728985    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728988    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728991    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728993    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728996    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.728998    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729001    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729003    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729006    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729009    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.743366 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729011    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729014    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729016    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729019    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729021    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729024    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729026    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729029    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729031    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729034    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729036    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729039    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729047    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729050    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729052    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729055    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729058    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729061    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729064    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729066    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.743876 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729069    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729073    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729078    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729080    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729084    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729087    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729089    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729092    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729094    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729097    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729099    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729102    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729105    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729107    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729110    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729112    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729115    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729118    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729120    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729123    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.744356 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729126    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729128    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729131    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729137    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729140    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729148    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729150    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729153    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729156    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729158    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729161    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729163    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729166    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729168    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729171    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729174    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729176    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729179    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729181    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729184    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.745117 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729186    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729189    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729191    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729194    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729196    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729199    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729201    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729206    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729209    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729212    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729215    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729217    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729220    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729222    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729225    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729227    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729230    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729232    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729240    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.745618 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729243    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729245    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.729269    4550 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729314    4550 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729318    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729321    4550 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729324    4550 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729327    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729330    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729333    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729335    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729338    4550 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729340    4550 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729343    4550 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729346    4550 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729349    4550 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729351    4550 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:05:17.746082 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729353    4550 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729356    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729358    4550 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729361    4550 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729364    4550 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729366    4550 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729369    4550 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729371    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729374    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729376    4550 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729379    4550 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729381    4550 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729384    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729386    4550 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729389    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729398    4550 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729401    4550 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729404    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729406    4550 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:05:17.746549 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729409    4550 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729411    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729415    4550 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729418    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729421    4550 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729424    4550 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729426    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729449    4550 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729452    4550 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729454    4550 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729457    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729460    4550 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729462    4550 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729465    4550 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729468    4550 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729471    4550 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729473    4550 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729479    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729482    4550 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:05:17.747054 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729485    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729487    4550 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729490    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729493    4550 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729495    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729498    4550 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729502    4550 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729505    4550 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729507    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729510    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729519    4550 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729521    4550 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729524    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729527    4550 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729529    4550 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729532    4550 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729535    4550 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729538    4550 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729540    4550 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:05:17.747979 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729544    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729547    4550 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729549    4550 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729552    4550 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729555    4550 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729557    4550 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729560    4550 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729562    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729565    4550 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729568    4550 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729570    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729574    4550 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729577    4550 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729579    4550 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729582    4550 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729585    4550 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729587    4550 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729590    4550 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729592    4550 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729596    4550 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:05:17.748652 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729599    4550 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729602    4550 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729604    4550 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729607    4550 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.729656    4550 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730765    4550 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730784    4550 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730790    4550 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730796    4550 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730801    4550 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730806    4550 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730811    4550 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730815    4550 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730820    4550 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:17.730825    4550 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.731136    4550 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.732210    4550 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.732276    4550 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 00:05:17.749295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.732484    4550 server.go:1014] "Starting client certificate rotation"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.732619    4550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.732665    4550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:37 +0000 UTC" deadline="2028-08-11 12:37:28.212750474 +0000 UTC"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.732695    4550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15540h32m10.480057062s"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.733019    4550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.734265    4550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.735938    4550 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.737673    4550 log.go:25] "Validated CRI v1 image API"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.738536    4550 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:05:17.749807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.740195    4550 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80e57c27-4e91-4c5b-b31a-0d2221eb3b9c:/dev/nvme0n1p4 a489c620-25f3-4da9-9a71-7b177d94d869:/dev/nvme0n1p3]
Nov 03 00:05:17.750042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.740217    4550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d97ffe5a33c1192bdd09894beb44be107315be716bb756de0d0f596aa5bf919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d97ffe5a33c1192bdd09894beb44be107315be716bb756de0d0f596aa5bf919/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89cef0df2aef88f17850600d694119afb4f0ad0e63afa81e6dad72c1f2df76ee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89cef0df2aef88f17850600d694119afb4f0ad0e63afa81e6dad72c1f2df76ee/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc6e2a29c3ab00719834c870960ea273378444c579e91a1e2b64e608531d77c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc6e2a29c3ab00719834c870960ea273378444c579e91a1e2b64e608531d77c8/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d387007-75e5-4186-8039-b96c77d34f1f/volumes/kubernetes.io~projected/kube-api-access-4hnz4:{mountpoint:/var/lib/kubelet/pods/0d387007-75e5-4186-8039-b96c77d34f1f/volumes/kubernetes.io~projected/kube-api-access-4hnz4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24c625e0-13a7-48df-bd0e-c383ff56ecca/volumes/kubernetes.io~projected/kube-api-access-98gwt:{mountpoint:/var/lib/kubelet/pods/24c625e0-13a7-48df-bd0e-c383ff56ecca/volumes/kubernetes.io~projected/kube-api-access-98gwt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b2e7c3b-981c-436b-a1cb-563254e2b11f/volumes/kubernetes.io~projected/kube-api-access-cf2kg:{mountpoint:/var/lib/kubelet/pods/4b2e7c3b-981c-436b-a1cb-563254e2b11f/volumes/kubernetes.io~projected/kube-api-access-cf2kg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d168306-103a-47b1-9b32-0c33b7825416/volumes/kubernetes.io~projected/kube-api-access-dgzfs:{mountpoint:/var/lib/kubelet/pods/4d168306-103a-47b1-9b32-0c33b7825416/volumes/kubernetes.io~projected/kube-api-access-dgzfs major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d168306-103a-47b1-9b32-0c33b7825416/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4d168306-103a-47b1-9b32-0c33b7825416/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~projected/kube-api-access-7frwz:{mountpoint:/var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~projected/kube-api-access-7frwz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6b2892-309f-49f9-94e2-9af7a736d740/volumes/kubernetes.io~projected/kube-api-access-btczm:{mountpoint:/var/lib/kubelet/pods/5f6b2892-309f-49f9-94e2-9af7a736d740/volumes/kubernetes.io~projected/kube-api-access-btczm major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f6b2892-309f-49f9-94e2-9af7a736d740/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/5f6b2892-309f-49f9-94e2-9af7a736d740/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a523c36-7d66-4549-b5fd-a9bc4903f53f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6a523c36-7d66-4549-b5fd-a9bc4903f53f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80f326a9-e6d5-45df-b03d-37d6bec25926/volumes/kubernetes.io~projected/kube-api-access-5g6kh:{mountpoint:/var/lib/kubelet/pods/80f326a9-e6d5-45df-b03d-37d6bec25926/volumes/kubernetes.io~projected/kube-api-access-5g6kh major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/821da046-5713-4974-ad87-8debd07cc8bf/volumes/kubernetes.io~projected/kube-api-access-smg86:{mountpoint:/var/lib/kubelet/pods/821da046-5713-4974-ad87-8debd07cc8bf/volumes/kubernetes.io~projected/kube-api-access-smg86 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/887b3093-6795-4b74-aa85-f46f3dbc76fd/volumes/kubernetes.io~projected/kube-api-access-9mksl:{mountpoint:/var/lib/kubelet/pods/887b3093-6795-4b74-aa85-f46f3dbc76fd/volumes/kubernetes.io~projected/kube-api-access-9mksl major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92908e9f-c806-49f8-8545-b4f2ff364cc4/volumes/kubernetes.io~projected/kube-api-access-v2hjp:{mountpoint:/var/lib/kubelet/pods/92908e9f-c806-49f8-8545-b4f2ff364cc4/volumes/kubernetes.io~projected/kube-api-access-v2hjp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0405b2f-5102-4390-a855-a14f0cad58e5/volumes/kubernetes.io~projected/kube-api-access-p49qh:{mountpoint:/var/lib/kubelet/pods/a0405b2f-5102-4390-a855-a14f0cad58e5/volumes/kubernetes.io~projected/kube-api-access-p49qh major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a91bc30d-2c15-48ab-97c9-c54713750b36/volumes/kubernetes.io~projected/kube-api-access-fxtx5:{mountpoint:/var/lib/kubelet/pods/a91bc30d-2c15-48ab-97c9-c54713750b36/volumes/kubernetes.io~projected/kube-api-access-fxtx5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e/volumes/kubernetes.io~projected/kube-api-access-c5vhh:{mountpoint:/var/lib/kubelet/pods/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e/volumes/kubernetes.io~projected/kube-api-access-c5vhh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5b7233c-5ec0-47fb-9129-793e2383b1bc/volumes/kubernetes.io~projected/kube-api-access-mj2sx:{mountpoint:/var/lib/kubelet/pods/d5b7233c-5ec0-47fb-9129-793e2383b1bc/volumes/kubernetes.io~projected/kube-api-access-mj2sx major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volumes/kubernetes.io~projected/kube-api-access-l8q6t:{mountpoint:/var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volumes/kubernetes.io~projected/kube-api-access-l8q6t major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9c9783e-067c-456e-b511-e687ca741b33/volumes/kubernetes.io~projected/kube-api-access-f4gsk:{mountpoint:/var/lib/kubelet/pods/f9c9783e-067c-456e-b511-e687ca741b33/volumes/kubernetes.io~projected/kube-api-access-f4gsk major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e39d268f7508699b0cb2f7978b859810541205b7cf06757d89e75c445df59bc9/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/eca547e1e5419941b55f25d50124612cfd4117bd257e4cfb3fab0d8378cc2e1d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3674cff0b94d2e9e1b15d71a482eccd57217d9c4b101e2cc3d3b510cfb4f5892/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f8413e501e6569fae442d3e050ae44d479c34947bd731c35883566be7cb308b3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e2d3a30618f0f34b6d8c1fed0dfd46cdd96ba8190cad50916e099ee92de744f4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1fed6f5c743129d4139b9e4d29f7382a078ed37a8ae0abee01d676f6e06de8f9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/00b39a7ee1df6edfccf731e6b50430ce8284f09fea0ba652905d9f37b6192952/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/46a55455408bdc093df2d5b215ae98445a32f01798a6bebd1a1e8d168ba8fd17/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/61247f0a9bcd566dd29fde08ece75351a26e4e7a3373fc7359a5bf378a65d191/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/23693c6d4ac3e2c97c60b25a6cf0f901a4e5e3cbc7c0cb3968f078cd77dc6b37/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/76ca4e885376ee2922878678dfa8e31a43bb5e21d86ae7ddc87623fc1729881d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/ad58de2702537b3b7a41d3ddda14b15aa30856bddfd40b7f282c9a070b97acee/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/63e9803d2a8989b391e74a4a846b9f8523be1a7cc7809208c7494fa51d87d2fe/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/6008d954ecf540ae42c70565dabea0f283b89c2f36f8b071d3d3577908229c1f/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/077ac536fb32191aa343fb9c15f99b58ca0ad605500e41dda2cba7d13ed7f1d3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0666450c093a902a958a90ea0152c21bf586739018c81aa066abda8d3b9a1ef4/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d749e2dd96736f2f3011840d8d26ff9d949488c26fa08e81fdbe449dbe60d8c3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/96195c5d64ff847e42baf09e314766b29585e23323e5956fbd6115f6ce4e44e9/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5d635a5f97684cce97d77741f36db045e19ab9cb9bede3d4614237937b180814/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/88b6cf20279dfe5377620610a9f2d4207b5edd44633412a881f72b9146d6d560/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/934a0551da6b5d463a7c92bb63271adf1171013f98a466f161d068d60f33692b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/c14e99148808f9e1cccef42f745c6979e73850635329fde4bf0cc87310bba2a1/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/73ccc912ff884dca7515416fa50f7863f81f96f2c3eafa536ebd3a3bb6e8ed19/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/5e2eb6e566d981234ae9dd0a815f535cf6e0d6fdf2c21e6fe00108f1e66ba6f6/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/c5b9642a96c4db67d54da603eb97094b8745edd59e7ef62db64781746a6c52dc/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/094b7df0278fd421e29909c43bdc9101bbcc2afc6b1c916f0fef4843ce439d70/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/55ad5657f70f0e6121d151bf04218ad896768d9d38cd6015f2f9783968cea3d5/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/73bd02ab58531ee9ff51f46bad3a19ea3cabf908ecb6b8988e032f875fc86042/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/4b1da5ea3803f5f8eba247ca4d462fba9285ed7b34a1c7893bcfbf538d2764d9/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/922649146a7512eca7c750352f7644960b3b5b76f5e5feb540b24929943412be/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/7fbe2a627aa685782fb63d99d0c8a7157744ca1ae1e64072e84244d476533cf0/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/1e0258d13ef37a6085735ab7de360d70af287e9ac0625a38888b226247c65fbc/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/687d49ebfb4856926fc0ff892e0e01280832042f3c5356724ef0b5497bddc5b0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/a7184da69d3564e7e91cf5488a4e4a6c5b3dd15cafaab6da2aeb37f7fc814906/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/795804245ec3ff3b04a75ebefa72565f751aa3d7526ee9362589ae1a1d2a3b10/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/6f98a3bfac032fd01c3a8618d9ee44bfd4f0916042dd447db417e77a884c3e61/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/07d3a10cab946c1aaefb6abe65119d1490ef0485f1e754998670ee4ecd28e5d0/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0fd2dd74001f125fa5ddc1c539de5a987415f22a4240bf70c55c7095c26597f9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5a732f6aa51df1710a45dd7c2c1bd0d638b8d67ba6464c602e37fbb24a6a5f0c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/ee24a04833425432d723ad52702e7819b8b0475e67731f9fc381d6dbd3c6e185/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f2ad40e4514384e64e486a1e579e438f2ab51020ba2323ec2634ad6299c82bbe/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/963d87e6ed9abed956ae925220b7fc757f8fb90f3eecf16932f80ef69cf76aef/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.746608    4550 manager.go:217] Machine: {Timestamp:2026-11-03 00:05:17.746103926 +0000 UTC m=+0.067640717 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098243 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29efc9ee0af4b493ab04a3f3da813f SystemUUID:ec29efc9-ee0a-f4b4-93ab-04a3f3da813f BootID:c43d4445-44af-4843-b038-ad126b028d21 Filesystems:[{Device:/run/containers/storage/overlay-containers/cf50793fc8022984d4ff00b7486511450f5ba6f884aa999056de96f8b9f09e6c/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6b2892-309f-49f9-94e2-9af7a736d740/volumes/kubernetes.io~projected/kube-api-access-btczm DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9e275dc3058a7f3161d8c45e1d23facfbfd37ff507cacdce9c7530e5fbf47e7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5f6b2892-309f-49f9-94e2-9af7a736d740/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0d387007-75e5-4186-8039-b96c77d34f1f/volumes/kubernetes.io~projected/kube-api-access-4hnz4 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a523c36-7d66-4549-b5fd-a9bc4903f53f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f9c9783e-067c-456e-b511-e687ca741b33/volumes/kubernetes.io~projected/kube-api-access-f4gsk DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5b7233c-5ec0-47fb-9129-793e2383b1bc/volumes/kubernetes.io~projected/kube-api-access-mj2sx DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a91bc30d-2c15-48ab-97c9-c54713750b36/volumes/kubernetes.io~projected/kube-api-access-fxtx5 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80f326a9-e6d5-45df-b03d-37d6bec25926/volumes/kubernetes.io~projected/kube-api-access-5g6kh DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/821da046-5713-4974-ad87-8debd07cc8bf/volumes/kubernetes.io~projected/kube-api-access-smg86 DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92908e9f-c806-49f8-8545-b4f2ff364cc4/volumes/kubernetes.io~projected/kube-api-access-v2hjp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d97ffe5a33c1192bdd09894beb44be107315be716bb756de0d0f596aa5bf919/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volumes/kubernetes.io~projected/kube-api-access-l8q6t DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bea24e1a5901bf0d4223f361b2df7c415b86322a1adffa76aca9436c7bae90d0/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4f2fbd15924e63a0d5cf5a58a0e2914d1d071d498c6a6bb5731b9c9a6ef5f39/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89cef0df2aef88f17850600d694119afb4f0ad0e63afa81e6dad72c1f2df76ee/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d168306-103a-47b1-9b32-0c33b7825416/volumes/kubernetes.io~projected/kube-api-access-dgzfs DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a0405b2f-5102-4390-a855-a14f0cad58e5/volumes/kubernetes.io~projected/kube-api-access-p49qh DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~projected/kube-api-access-7frwz DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/853c0821aac58fdafd2cda74269b1f2b520e45e50ce0a153e326c1f57373a4f7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d168306-103a-47b1-9b32-0c33b7825416/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2faa835b7930def1b19404b10b567894c08267dfe4b83594e790562dd070a9ca/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b118c1e4e3e5ded22296fa99422b04c2bf1aa23820e1802d9058fe04215a064/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/5c2280c9-7198-40c7-b6ac-fd299946a0d5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/887b3093-6795-4b74-aa85-f46f3dbc76fd/volumes/kubernetes.io~projected/kube-api-access-9mksl DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1666d366eded497788584acff2c4ef9ae75b0e3077bf1a0db4126a19174c34f/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfcef7a57257d4767f02d4bf66c1271f5dbffd157ef18bbd99624f4ce1038c3d/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24c625e0-13a7-48df-bd0e-c383ff56ecca/volumes/kubernetes.io~projected/kube-api-access-98gwt DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e1479ca0-d25e-42a0-96a4-877e39edcdbb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b35eeb0560d68bf48ef028d58f767e9c4679aa84f7687748f48b99854fb7118e/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc6e2a29c3ab00719834c870960ea273378444c579e91a1e2b64e608531d77c8/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e/volumes/kubernetes.io~projected/kube-api-access-c5vhh DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b2e7c3b-981c-436b-a1cb-563254e2b11f/volumes/kubernetes.io~projected/kube-api-access-cf2kg DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2faa835b7930def MacAddress:6e:18:f3:97:c8:ce Speed:10000 Mtu:8901} {Name:5b118c1e4e3e5de MacAddress:b6:b4:7f:0c:6e:cb Speed:10000 Mtu:8901} {Name:a4f2fbd15924e63 MacAddress:ee:2b:38:75:9b:7f Speed:10000 Mtu:8901} {Name:b35eeb0560d68bf MacAddress:22:05:4c:be:5f:d0 Speed:10000 Mtu:8901} {Name:bc6e2a29c3ab007 MacAddress:5e:1e:75:26:d3:5a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d9:80:5f:29 Speed:0 Mtu:9001} {Name:br-int MacAddress:72:67:f4:fe:fc:b1 Speed:0 Mtu:8901} {Name:df22a5c6877e95c MacAddress:86:9d:0b:db:d9:b7 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d9:80:5f:29 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:4e:a1:1f:6c:cd:41 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:4d:cd:51:e8:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.747061    4550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.747160    4550 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.747605    4550 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.747636    4550 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.748071    4550 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.748084    4550 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.748100    4550 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.748133    4550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.748273    4550 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.748976    4550 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749104    4550 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749144    4550 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749161    4550 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749171    4550 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749180    4550 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749188    4550 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.749994    4550 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:05:17.750205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.750010    4550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:05:17.751099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.750646    4550 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:05:17.759321 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759300    4550 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:05:17.759321 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759318    4550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:05:17.759501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759488    4550 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 00:05:17.759720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759704    4550 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:05:17.759803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759744    4550 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:05:17.759905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759892    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:05:17.759964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759921    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:05:17.759964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759933    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:05:17.759964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759942    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:05:17.759964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759951    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:05:17.759964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759960    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759970    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759979    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.759992    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.760002    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.760023    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.760039    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.760075    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:05:17.760190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.760085    4550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:05:17.763670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.763655    4550 server.go:1263] "Started kubelet"
Nov 03 00:05:17.763787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.763751    4550 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:05:17.764616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.763944    4550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:05:17.764616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.764310    4550 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:05:17.764203 ip-10-0-133-253 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:05:17.765037 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.764915    4550 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:05:17.765838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.765820    4550 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:05:17.769764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.769739    4550 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:05:17.770628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.770605    4550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:05:17.771496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.771028    4550 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:05:17.771496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.771043    4550 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:05:17.771496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.771063    4550 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:05:17.771496 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:17.771273    4550 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-253.ec2.internal\" not found"
Nov 03 00:05:17.772095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.772018    4550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:39 +0000 UTC" deadline="2028-05-31 00:01:45.048034662 +0000 UTC"
Nov 03 00:05:17.772095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.772095    4550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13799h56m27.275944289s"
Nov 03 00:05:17.773666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.773646    4550 factory.go:55] Registering systemd factory
Nov 03 00:05:17.773666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.773667    4550 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:05:17.774021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774002    4550 factory.go:153] Registering CRI-O factory
Nov 03 00:05:17.774319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774297    4550 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:05:17.774398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774371    4550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:05:17.774398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774396    4550 factory.go:103] Registering Raw factory
Nov 03 00:05:17.774539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774410    4550 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:05:17.774596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774542    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1479ca0-d25e-42a0-96a4-877e39edcdbb" volumeName="kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 00:05:17.774657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774593    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1479ca0-d25e-42a0-96a4-877e39edcdbb" volumeName="kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-config" seLinuxMountContext=""
Nov 03 00:05:17.774657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774607    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9c9783e-067c-456e-b511-e687ca741b33" volumeName="kubernetes.io/projected/f9c9783e-067c-456e-b511-e687ca741b33-kube-api-access-f4gsk" seLinuxMountContext=""
Nov 03 00:05:17.774657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774650    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6b2892-309f-49f9-94e2-9af7a736d740" volumeName="kubernetes.io/projected/5f6b2892-309f-49f9-94e2-9af7a736d740-kube-api-access-btczm" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774660    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80f326a9-e6d5-45df-b03d-37d6bec25926" volumeName="kubernetes.io/projected/80f326a9-e6d5-45df-b03d-37d6bec25926-kube-api-access-5g6kh" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774669    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="821da046-5713-4974-ad87-8debd07cc8bf" volumeName="kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774676    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="821da046-5713-4974-ad87-8debd07cc8bf" volumeName="kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774684    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a91bc30d-2c15-48ab-97c9-c54713750b36" volumeName="kubernetes.io/projected/a91bc30d-2c15-48ab-97c9-c54713750b36-kube-api-access-fxtx5" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774694    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d387007-75e5-4186-8039-b96c77d34f1f" volumeName="kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-multus-daemon-config" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774704    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d387007-75e5-4186-8039-b96c77d34f1f" volumeName="kubernetes.io/projected/0d387007-75e5-4186-8039-b96c77d34f1f-kube-api-access-4hnz4" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774712    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5b7233c-5ec0-47fb-9129-793e2383b1bc" volumeName="kubernetes.io/projected/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-api-access-mj2sx" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774718    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1479ca0-d25e-42a0-96a4-877e39edcdbb" volumeName="kubernetes.io/projected/e1479ca0-d25e-42a0-96a4-877e39edcdbb-kube-api-access-l8q6t" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774725    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1479ca0-d25e-42a0-96a4-877e39edcdbb" volumeName="kubernetes.io/secret/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774732    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d387007-75e5-4186-8039-b96c77d34f1f" volumeName="kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774742    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b2e7c3b-981c-436b-a1cb-563254e2b11f" volumeName="kubernetes.io/configmap/4b2e7c3b-981c-436b-a1cb-563254e2b11f-iptables-alerter-script" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774749    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d168306-103a-47b1-9b32-0c33b7825416" volumeName="kubernetes.io/secret/4d168306-103a-47b1-9b32-0c33b7825416-cert" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774758    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c2280c9-7198-40c7-b6ac-fd299946a0d5" volumeName="kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-tuned" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774765    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0405b2f-5102-4390-a855-a14f0cad58e5" volumeName="kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774772    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a91bc30d-2c15-48ab-97c9-c54713750b36" volumeName="kubernetes.io/empty-dir/a91bc30d-2c15-48ab-97c9-c54713750b36-tmp-dir" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774779    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab4f37ba-6131-49ba-bd2a-7ecf1dded60e" volumeName="kubernetes.io/projected/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kube-api-access-c5vhh" seLinuxMountContext=""
Nov 03 00:05:17.774802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774783    4550 manager.go:319] Starting recovery of all containers
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774786    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5b7233c-5ec0-47fb-9129-793e2383b1bc" volumeName="kubernetes.io/empty-dir/d5b7233c-5ec0-47fb-9129-793e2383b1bc-data-volume" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.774993    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c625e0-13a7-48df-bd0e-c383ff56ecca" volumeName="kubernetes.io/configmap/24c625e0-13a7-48df-bd0e-c383ff56ecca-config-volume" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775041    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d168306-103a-47b1-9b32-0c33b7825416" volumeName="kubernetes.io/projected/4d168306-103a-47b1-9b32-0c33b7825416-kube-api-access-dgzfs" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775056    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c2280c9-7198-40c7-b6ac-fd299946a0d5" volumeName="kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-tmp" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775071    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="821da046-5713-4974-ad87-8debd07cc8bf" volumeName="kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775109    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92908e9f-c806-49f8-8545-b4f2ff364cc4" volumeName="kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775122    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3" volumeName="kubernetes.io/configmap/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-konnectivity-ca" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775136    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3" volumeName="kubernetes.io/secret/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-agent-certs" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775190    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1479ca0-d25e-42a0-96a4-877e39edcdbb" volumeName="kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-env-overrides" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775213    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c625e0-13a7-48df-bd0e-c383ff56ecca" volumeName="kubernetes.io/empty-dir/24c625e0-13a7-48df-bd0e-c383ff56ecca-tmp-dir" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775251    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b2e7c3b-981c-436b-a1cb-563254e2b11f" volumeName="kubernetes.io/projected/4b2e7c3b-981c-436b-a1cb-563254e2b11f-kube-api-access-cf2kg" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775262    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6b2892-309f-49f9-94e2-9af7a736d740" volumeName="kubernetes.io/secret/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-key" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775274    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a523c36-7d66-4549-b5fd-a9bc4903f53f" volumeName="kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775286    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b3093-6795-4b74-aa85-f46f3dbc76fd" volumeName="kubernetes.io/configmap/887b3093-6795-4b74-aa85-f46f3dbc76fd-serviceca" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775331    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="887b3093-6795-4b74-aa85-f46f3dbc76fd" volumeName="kubernetes.io/projected/887b3093-6795-4b74-aa85-f46f3dbc76fd-kube-api-access-9mksl" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775346    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24c625e0-13a7-48df-bd0e-c383ff56ecca" volumeName="kubernetes.io/projected/24c625e0-13a7-48df-bd0e-c383ff56ecca-kube-api-access-98gwt" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775356    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f6b2892-309f-49f9-94e2-9af7a736d740" volumeName="kubernetes.io/configmap/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-cabundle" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775387    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c2280c9-7198-40c7-b6ac-fd299946a0d5" volumeName="kubernetes.io/projected/5c2280c9-7198-40c7-b6ac-fd299946a0d5-kube-api-access-7frwz" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775396    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="821da046-5713-4974-ad87-8debd07cc8bf" volumeName="kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775406    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92908e9f-c806-49f8-8545-b4f2ff364cc4" volumeName="kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775413    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92908e9f-c806-49f8-8545-b4f2ff364cc4" volumeName="kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775421    4550 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92908e9f-c806-49f8-8545-b4f2ff364cc4" volumeName="kubernetes.io/projected/92908e9f-c806-49f8-8545-b4f2ff364cc4-kube-api-access-v2hjp" seLinuxMountContext=""
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775444    4550 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:05:17.775613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.775453    4550 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:05:17.798288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.798238    4550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:05:17.799764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.799744    4550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:05:17.799764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.799768    4550 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:05:17.799948 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.799787    4550 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:05:17.799948 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:17.799818    4550 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 00:05:17.808173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.808148    4550 generic.go:356] "Generic (PLEG): container finished" podID="09e361fc4461f10d60238f71bbc39227" containerID="8a2f9beb06675417af14d4bffc20de8b6611af485012957088df0a440ab50f7f" exitCode=0
Nov 03 00:05:17.810058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.810035    4550 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.810255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.810231    4550 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 00:05:17.810380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.810342    4550 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.823454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.823418    4550 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.823607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.823584    4550 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:05:17.834138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.833965    4550 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="d450a71606b75862d0ab3aa73ef4a96084b2e977b6c1dd98510ec4e070cbdb79" exitCode=0
Nov 03 00:05:17.834293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.834144    4550 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="6a70306f11c060842e54c7b84c9bab9e46339a7453dbe1dad8e98c1145560d4d" exitCode=0
Nov 03 00:05:17.834293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.834155    4550 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="545f8240e7d0111e7d9c8d349340afa128e806e40568165480f9080ee1630816" exitCode=0
Nov 03 00:05:17.834293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.834163    4550 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="9e1da9ef05a98cfdfa9008a992e094b5ea213f024d00ab5de93c32a93c9d6fc6" exitCode=0
Nov 03 00:05:17.834293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.834171    4550 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="d0ac352adbcedccf65b340e20bd89e7ca8dac7ff10e49bc180f92ec4a7f88e67" exitCode=0
Nov 03 00:05:17.834293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.834178    4550 generic.go:356] "Generic (PLEG): container finished" podID="92908e9f-c806-49f8-8545-b4f2ff364cc4" containerID="06705207f4a19fe3f293ff72786e66f8dce58a52fad94b2c024923dbb9f9c98a" exitCode=0
Nov 03 00:05:17.843407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.843360    4550 manager.go:324] Recovery completed
Nov 03 00:05:17.857744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857724    4550 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:05:17.857744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857737    4550 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:05:17.857875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857753    4550 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:05:17.857905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857891    4550 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 00:05:17.857934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857902    4550 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 00:05:17.857934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857917    4550 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 00:05:17.857934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857923    4550 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 00:05:17.857934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857928    4550 policy_none.go:50] "Start"
Nov 03 00:05:17.858039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857949    4550 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:05:17.858039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.857958    4550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:05:17.858089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.858077    4550 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 00:05:17.858089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.858084    4550 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 00:05:17.858089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.858089    4550 policy_none.go:44] "Start"
Nov 03 00:05:17.862559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.862062    4550 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:05:17.867333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867317    4550 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:05:17.867374 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:17.867359    4550 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:05:17.867374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867368    4550 server.go:87] "Starting device plugin registration server"
Nov 03 00:05:17.867645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867632    4550 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:05:17.867706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867644    4550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:05:17.867789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867763    4550 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:05:17.867912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867878    4550 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:05:17.867966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867919    4550 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:05:17.867966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.867927    4550 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:05:17.868105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.868014    4550 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:05:17.867907558Z","UUID":"a86016da-177d-4e46-aadb-c472410afcc1","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:05:17.869033 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:17.869012    4550 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:05:17.869291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.869276    4550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:05:17.869355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.869302    4550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:05:17.900534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.900477    4550 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"]
Nov 03 00:05:17.901013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.900977    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal" event={"ID":"e107905b7a9295ebc233864ba55f6a44","Type":"ContainerStarted","Data":"f1f455723cee090719c00f297b2b9476407415ebfe916a190f489976c2651fbe"}
Nov 03 00:05:17.901122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.901023    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal" event={"ID":"e107905b7a9295ebc233864ba55f6a44","Type":"ContainerStarted","Data":"ad93c03000fae52d3d6965448f0e5b2d3d5c82d5de9f8c1fa02078031c03a826"}
Nov 03 00:05:17.901122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.901043    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" event={"ID":"09e361fc4461f10d60238f71bbc39227","Type":"ContainerStarted","Data":"2e268be7f92dfea96d9aefc9ab7d29e5d3dedc76f4f633866f288d975c336850"}
Nov 03 00:05:17.901122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.901062    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" event={"ID":"09e361fc4461f10d60238f71bbc39227","Type":"ContainerDied","Data":"8a2f9beb06675417af14d4bffc20de8b6611af485012957088df0a440ab50f7f"}
Nov 03 00:05:17.901122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.901084    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal" event={"ID":"09e361fc4461f10d60238f71bbc39227","Type":"ContainerStarted","Data":"43260e55020bf62da456c45a797a2cd96e95830e6d70d8621d8100f318ced7c7"}
Nov 03 00:05:17.901271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.901199    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:17.901306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.901202    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:17.925918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.925893    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:17.926046 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:17.925940    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:17.926489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.926469    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:17.926560 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:17.926513    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:17.968612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.968573    4550 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:05:17.970026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.970006    4550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:05:17.970110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.970041    4550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:05:17.970110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.970052    4550 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-253.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:05:17.970110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:17.970092    4550 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.005420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.005393    4550 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.005503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.005481    4550 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.010506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.010481    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.010620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.010513    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.010620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.010536    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e107905b7a9295ebc233864ba55f6a44-config\") pod \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" (UID: \"e107905b7a9295ebc233864ba55f6a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.111045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.110963    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.111045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.110912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.111226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.111043    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.111226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.111107    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e107905b7a9295ebc233864ba55f6a44-config\") pod \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" (UID: \"e107905b7a9295ebc233864ba55f6a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.111226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.111119    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09e361fc4461f10d60238f71bbc39227-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" (UID: \"09e361fc4461f10d60238f71bbc39227\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.111226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.111145    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e107905b7a9295ebc233864ba55f6a44-config\") pod \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" (UID: \"e107905b7a9295ebc233864ba55f6a44\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.749656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.749623    4550 apiserver.go:52] "Watching apiserver"
Nov 03 00:05:18.804191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.804154    4550 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:05:18.804719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.804692    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-vlgrc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq","openshift-ingress-canary/ingress-canary-sjlsx","openshift-dns/dns-default-pzqlp","openshift-multus/multus-additional-cni-plugins-v7x7d","openshift-cluster-node-tuning-operator/tuned-tddjp","kube-system/global-pull-secret-syncer-9rxv7","kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j","openshift-ovn-kubernetes/ovnkube-node-tnptp","openshift-service-ca/service-ca-5fffc97bdf-drrmw","openshift-dns/node-resolver-5jkfg","openshift-network-operator/iptables-alerter-s2gsr","openshift-multus/network-metrics-daemon-t4t4w","openshift-image-registry/node-ca-96xx2","openshift-insights/insights-runtime-extractor-4cnb6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal","openshift-multus/multus-7dkm6","openshift-insights/periodic-gathering-8whn6-nj4vj","kube-system/konnectivity-agent-4zf7k"]
Nov 03 00:05:18.811046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.811021    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:18.812398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.812379    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.813145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.813130    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.820809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.820786    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:05:18.820906 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.820817    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:05:18.820906 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.820865    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nrnbx\""
Nov 03 00:05:18.821555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.821540    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:05:18.823074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823057    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:05:18.823155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823093    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.823155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823113    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gt4zj\""
Nov 03 00:05:18.823155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823134    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.823155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823152    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:05:18.823401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823389    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:05:18.823851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.823839    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:05:18.824111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.824100    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.824150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.824115    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.824150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.824143    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.831662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.831647    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jtdbf\""
Nov 03 00:05:18.831838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.831824    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9mnsd\""
Nov 03 00:05:18.831838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.831834    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xfsv5\""
Nov 03 00:05:18.832806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.832791    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:05:18.832920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.832905    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:05:18.833110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.833097    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:05:18.833303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.833288    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.833354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.833341    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-k88mc\""
Nov 03 00:05:18.833403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.833361    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.833403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.833396    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ct6nh\""
Nov 03 00:05:18.834303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.834290    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:05:18.834617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.834602    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:05:18.834678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.834605    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 00:05:18.834678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.834605    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4v7ss\""
Nov 03 00:05:18.835054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835033    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.835054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835060    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qr8z8\""
Nov 03 00:05:18.835523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835465    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.835523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835517    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:05:18.835672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835650    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:05:18.835774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835717    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7llq5\""
Nov 03 00:05:18.835774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835734    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzkq\""
Nov 03 00:05:18.835856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835800    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:05:18.835856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835816    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.835941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835883    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rns5k\""
Nov 03 00:05:18.835941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835821    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.835941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835894    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835946    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.836053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835979    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:05:18.836053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.835468    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.836053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836009    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836149    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqxtt\""
Nov 03 00:05:18.836208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836178    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.836208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836198    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:05:18.836208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836202    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836226    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:05:18.836298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836251    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836471    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:05:18.836588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836569    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pr6ld\""
Nov 03 00:05:18.836703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836647    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836699    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:05:18.836802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836713    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836764    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:05:18.836802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836741    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.836802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836803    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836821    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 00:05:18.836940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836845    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 00:05:18.836940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836860    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:05:18.836940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836868    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:05:18.836940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836916    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7b8cw\""
Nov 03 00:05:18.837122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.836954    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:05:18.837315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.837297    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:05:18.838655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.838640    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.838810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.838797    4550 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.847793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.847763    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 00:05:18.862382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.862356    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:18.862532 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:18.862420    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-253.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.862950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.862937    4550 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:05:18.863028 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:18.862968    4550 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-253.ec2.internal"
Nov 03 00:05:18.872156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.872134    4550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:05:18.914771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.914741    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btczm\" (UniqueName: \"kubernetes.io/projected/5f6b2892-309f-49f9-94e2-9af7a736d740-kube-api-access-btczm\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:18.914771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.914774    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovn-node-metrics-cert\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.914987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.914797    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-kubernetes\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.914987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.914880    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-conf\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.914987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.914919    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:18.914987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.914958    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gsk\" (UniqueName: \"kubernetes.io/projected/f9c9783e-067c-456e-b511-e687ca741b33-kube-api-access-f4gsk\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:18.915197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/887b3093-6795-4b74-aa85-f46f3dbc76fd-serviceca\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:18.915197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915083    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-agent-certs\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:18.915197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915081    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovn-node-metrics-cert\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.915197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915156    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-multus\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.915197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915181    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-conf-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.915397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915285    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtx5\" (UniqueName: \"kubernetes.io/projected/a91bc30d-2c15-48ab-97c9-c54713750b36-kube-api-access-fxtx5\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:18.915397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915333    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-lib-modules\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.915397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915366    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hnz4\" (UniqueName: \"kubernetes.io/projected/0d387007-75e5-4186-8039-b96c77d34f1f-kube-api-access-4hnz4\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.915397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915375    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/887b3093-6795-4b74-aa85-f46f3dbc76fd-serviceca\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:18.915607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915413    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.915607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915481    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-socket-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.915607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915485    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-agent-certs\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:18.915607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915521    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-sys-fs\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.915607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915545    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.915607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915564    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.915863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915617    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-slash\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.915863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-systemd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.915863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915723    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.915863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915758    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-etc-selinux\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.915863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915812    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5vhh\" (UniqueName: \"kubernetes.io/projected/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kube-api-access-c5vhh\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.915863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915847    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-var-lib-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915871    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-run\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915881    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915906    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5b7233c-5ec0-47fb-9129-793e2383b1bc-crio-socket\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915936    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5b7233c-5ec0-47fb-9129-793e2383b1bc-data-volume\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915939    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.915963    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-cni-binary-copy\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916024    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-netd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916050    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5b7233c-5ec0-47fb-9129-793e2383b1bc-data-volume\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916092    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916120    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24c625e0-13a7-48df-bd0e-c383ff56ecca-config-volume\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916142    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-system-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916176    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-cni-binary-copy\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916182    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-kubelet\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916219    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b2e7c3b-981c-436b-a1cb-563254e2b11f-iptables-alerter-script\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:18.916283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916256    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916298    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/24c625e0-13a7-48df-bd0e-c383ff56ecca-config-volume\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916361    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916394    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916420    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-log-socket\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916471    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4b2e7c3b-981c-436b-a1cb-563254e2b11f-iptables-alerter-script\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916481    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:18.916551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916511    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-systemd\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916582    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-sys\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916610    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92908e9f-c806-49f8-8545-b4f2ff364cc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916626    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916660    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-systemd-units\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916698    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-bin\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916703    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.916744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916720    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-tmp\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916769    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916776    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-tmp\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-socket-dir-parent\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916810    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-netns\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916836    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-key\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916875    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-etc-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.916929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916893    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-script-lib\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.916972    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:18.917113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917046    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-key\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:18.917113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917084    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2kg\" (UniqueName: \"kubernetes.io/projected/4b2e7c3b-981c-436b-a1cb-563254e2b11f-kube-api-access-cf2kg\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:18.917113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917087    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-script-lib\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917118    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-cnibin\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.917234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917148    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c625e0-13a7-48df-bd0e-c383ff56ecca-tmp-dir\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.917234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917170    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-cnibin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.917234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917192    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-k8s-cni-cncf-io\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.917234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917205    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a523c36-7d66-4549-b5fd-a9bc4903f53f-original-pull-secret\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917245    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24c625e0-13a7-48df-bd0e-c383ff56ecca-tmp-dir\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917273    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-hostroot\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917309    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917379    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917395    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917411    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-ovn\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917466    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-modprobe-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.917493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917492    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-device-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917515    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-netns\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-node-log\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917567    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frwz\" (UniqueName: \"kubernetes.io/projected/5c2280c9-7198-40c7-b6ac-fd299946a0d5-kube-api-access-7frwz\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917601    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-bin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917727    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917767    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gwt\" (UniqueName: \"kubernetes.io/projected/24c625e0-13a7-48df-bd0e-c383ff56ecca-kube-api-access-98gwt\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917802    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-env-overrides\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917833    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mksl\" (UniqueName: \"kubernetes.io/projected/887b3093-6795-4b74-aa85-f46f3dbc76fd-kube-api-access-9mksl\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:18.917861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917862    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-var-lib-kubelet\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917885    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2sx\" (UniqueName: \"kubernetes.io/projected/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-api-access-mj2sx\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917939    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-config\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917959    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-registration-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917942    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-env-overrides\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.917989    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-kubelet-config\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918010    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a91bc30d-2c15-48ab-97c9-c54713750b36-hosts-file\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918032    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918097    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918119    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-kubelet\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918146    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e1479ca0-d25e-42a0-96a4-877e39edcdbb-ovnkube-config\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918184    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysconfig\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918206    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2e7c3b-981c-436b-a1cb-563254e2b11f-host-slash\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918215    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.918223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918228    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-cabundle\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918269    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2hjp\" (UniqueName: \"kubernetes.io/projected/92908e9f-c806-49f8-8545-b4f2ff364cc4-kube-api-access-v2hjp\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918288    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918304    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8q6t\" (UniqueName: \"kubernetes.io/projected/e1479ca0-d25e-42a0-96a4-877e39edcdbb-kube-api-access-l8q6t\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918394    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/887b3093-6795-4b74-aa85-f46f3dbc76fd-host\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918461    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-host\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918495    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d168306-103a-47b1-9b32-0c33b7825416-cert\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918512    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5f6b2892-309f-49f9-94e2-9af7a736d740-signing-cabundle\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918514    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6kh\" (UniqueName: \"kubernetes.io/projected/80f326a9-e6d5-45df-b03d-37d6bec25926-kube-api-access-5g6kh\") pod \"migrator-66df5d496b-dd54j\" (UID: \"80f326a9-e6d5-45df-b03d-37d6bec25926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918550    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-tuned\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918583    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgzfs\" (UniqueName: \"kubernetes.io/projected/4d168306-103a-47b1-9b32-0c33b7825416-kube-api-access-dgzfs\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918600    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-tuned\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918629    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918648    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-konnectivity-ca\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-multus-daemon-config\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918746    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-multus-certs\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.918793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918764    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d168306-103a-47b1-9b32-0c33b7825416-cert\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:18.919295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918810    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3-konnectivity-ca\") pod \"konnectivity-agent-4zf7k\" (UID: \"dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3\") " pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:18.919295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918853    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-etc-kubernetes\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.919295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918884    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-dbus\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:18.919295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d387007-75e5-4186-8039-b96c77d34f1f-multus-daemon-config\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.919295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918903    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91bc30d-2c15-48ab-97c9-c54713750b36-tmp-dir\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:18.919295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.918945    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a91bc30d-2c15-48ab-97c9-c54713750b36-tmp-dir\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:18.933255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.933224    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gsk\" (UniqueName: \"kubernetes.io/projected/f9c9783e-067c-456e-b511-e687ca741b33-kube-api-access-f4gsk\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:18.933739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.933717    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hnz4\" (UniqueName: \"kubernetes.io/projected/0d387007-75e5-4186-8039-b96c77d34f1f-kube-api-access-4hnz4\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:18.933739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.933730    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5vhh\" (UniqueName: \"kubernetes.io/projected/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kube-api-access-c5vhh\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:18.933849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.933742    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtx5\" (UniqueName: \"kubernetes.io/projected/a91bc30d-2c15-48ab-97c9-c54713750b36-kube-api-access-fxtx5\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:18.934376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.934361    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btczm\" (UniqueName: \"kubernetes.io/projected/5f6b2892-309f-49f9-94e2-9af7a736d740-kube-api-access-btczm\") pod \"service-ca-5fffc97bdf-drrmw\" (UID: \"5f6b2892-309f-49f9-94e2-9af7a736d740\") " pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw"
Nov 03 00:05:18.934525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.934510    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49qh\" (UniqueName: \"kubernetes.io/projected/a0405b2f-5102-4390-a855-a14f0cad58e5-kube-api-access-p49qh\") pod \"network-check-target-vlgrc\" (UID: \"a0405b2f-5102-4390-a855-a14f0cad58e5\") " pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:18.934660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.934646    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2kg\" (UniqueName: \"kubernetes.io/projected/4b2e7c3b-981c-436b-a1cb-563254e2b11f-kube-api-access-cf2kg\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:18.936711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.936674    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6kh\" (UniqueName: \"kubernetes.io/projected/80f326a9-e6d5-45df-b03d-37d6bec25926-kube-api-access-5g6kh\") pod \"migrator-66df5d496b-dd54j\" (UID: \"80f326a9-e6d5-45df-b03d-37d6bec25926\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-dd54j"
Nov 03 00:05:18.936711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.936692    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") pod \"periodic-gathering-8whn6-nj4vj\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") " pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:05:18.936862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.936712    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frwz\" (UniqueName: \"kubernetes.io/projected/5c2280c9-7198-40c7-b6ac-fd299946a0d5-kube-api-access-7frwz\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:18.936862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.936818    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mksl\" (UniqueName: \"kubernetes.io/projected/887b3093-6795-4b74-aa85-f46f3dbc76fd-kube-api-access-9mksl\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:18.936949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.936892    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2sx\" (UniqueName: \"kubernetes.io/projected/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-api-access-mj2sx\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:18.937051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.937038    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gwt\" (UniqueName: \"kubernetes.io/projected/24c625e0-13a7-48df-bd0e-c383ff56ecca-kube-api-access-98gwt\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:18.937207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.937190    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2hjp\" (UniqueName: \"kubernetes.io/projected/92908e9f-c806-49f8-8545-b4f2ff364cc4-kube-api-access-v2hjp\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:18.937563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.937548    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8q6t\" (UniqueName: \"kubernetes.io/projected/e1479ca0-d25e-42a0-96a4-877e39edcdbb-kube-api-access-l8q6t\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:18.937701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:18.937685    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgzfs\" (UniqueName: \"kubernetes.io/projected/4d168306-103a-47b1-9b32-0c33b7825416-kube-api-access-dgzfs\") pod \"ingress-canary-sjlsx\" (UID: \"4d168306-103a-47b1-9b32-0c33b7825416\") " pod="openshift-ingress-canary/ingress-canary-sjlsx"
Nov 03 00:05:19.019298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019210    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-lib-modules\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.019298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019254    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019278    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-socket-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.019298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019299    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-sys-fs\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019323    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-slash\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019344    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-systemd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019331    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019368    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019405    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-etc-selinux\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019415    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-slash\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019402    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-lib-modules\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019451    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-systemd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019425    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-sys-fs\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019477    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-var-lib-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019495    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-etc-selinux\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019500    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-run\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019533    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-socket-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019551    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-var-lib-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019545    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5b7233c-5ec0-47fb-9129-793e2383b1bc-crio-socket\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019524    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019588    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5b7233c-5ec0-47fb-9129-793e2383b1bc-crio-socket\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019591    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-netd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019621    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-netd\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.019629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019621    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019653    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019655    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-system-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019578    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-run\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019684    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-kubelet\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019701    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-system-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019713    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019750    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-kubelet\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019764    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019775    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-log-socket\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019796    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-systemd\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019834    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-log-socket\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019846    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-systemd\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019867    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-sys\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019894    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-sys\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019893    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019920    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-systemd-units\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019944    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-bin\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019967    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.019983    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-systemd-units\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020000    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-cni-bin\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020020    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-socket-dir-parent\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020039    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-cni-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020042    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-netns\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020066    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-netns\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020074    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-etc-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020084    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-socket-dir-parent\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020105    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-etc-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020118    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-cnibin\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020149    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-cnibin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020203    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-cnibin\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020220    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-k8s-cni-cncf-io\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020240    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-cnibin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020245    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-hostroot\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020264    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-k8s-cni-cncf-io\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020276    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020287    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-hostroot\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020300    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-ovn\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020310    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-openvswitch\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020323    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-modprobe-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020333    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-run-ovn\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020346    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-device-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020371    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-netns\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020395    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-node-log\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-modprobe-d\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020418    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-bin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020454    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-device-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020464    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-run-netns\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020464    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020483    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-node-log\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020490    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-var-lib-kubelet\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020498    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-bin\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020506    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92908e9f-c806-49f8-8545-b4f2ff364cc4-system-cni-dir\") pod \"multus-additional-cni-plugins-v7x7d\" (UID: \"92908e9f-c806-49f8-8545-b4f2ff364cc4\") " pod="openshift-multus/multus-additional-cni-plugins-v7x7d"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020508    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:19.020522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020554    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-var-lib-kubelet\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.020573    4550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020588    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-registration-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020617    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-kubelet-config\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020641    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a91bc30d-2c15-48ab-97c9-c54713750b36-hosts-file\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.020676    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.520654472 +0000 UTC m=+1.842191269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020687    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a91bc30d-2c15-48ab-97c9-c54713750b36-hosts-file\") pod \"node-resolver-5jkfg\" (UID: \"a91bc30d-2c15-48ab-97c9-c54713750b36\") " pod="openshift-dns/node-resolver-5jkfg"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020715    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-kubelet-config\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020730    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-kubelet\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020749    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysconfig\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020761    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1479ca0-d25e-42a0-96a4-877e39edcdbb-host-kubelet\") pod \"ovnkube-node-tnptp\" (UID: \"e1479ca0-d25e-42a0-96a4-877e39edcdbb\") " pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020730    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ab4f37ba-6131-49ba-bd2a-7ecf1dded60e-registration-dir\") pod \"aws-ebs-csi-driver-node-22gqq\" (UID: \"ab4f37ba-6131-49ba-bd2a-7ecf1dded60e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22gqq"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020775    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2e7c3b-981c-436b-a1cb-563254e2b11f-host-slash\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020796    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4b2e7c3b-981c-436b-a1cb-563254e2b11f-host-slash\") pod \"iptables-alerter-s2gsr\" (UID: \"4b2e7c3b-981c-436b-a1cb-563254e2b11f\") " pod="openshift-network-operator/iptables-alerter-s2gsr"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020812    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysconfig\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020814    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020837    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/887b3093-6795-4b74-aa85-f46f3dbc76fd-host\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020867    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-host\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020890    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020908    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-multus-certs\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020914    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/887b3093-6795-4b74-aa85-f46f3dbc76fd-host\") pod \"node-ca-96xx2\" (UID: \"887b3093-6795-4b74-aa85-f46f3dbc76fd\") " pod="openshift-image-registry/node-ca-96xx2"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020927    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-etc-kubernetes\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.020945    4550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020949    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-etc-kubernetes\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020964    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-dbus\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.020983    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-run-multus-certs\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.020995    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.520983548 +0000 UTC m=+1.842520353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.021000    4550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021007    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-kubernetes\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021014    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-host\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-conf\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021024    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a523c36-7d66-4549-b5fd-a9bc4903f53f-dbus\") pod \"global-pull-secret-syncer-9rxv7\" (UID: \"6a523c36-7d66-4549-b5fd-a9bc4903f53f\") " pod="kube-system/global-pull-secret-syncer-9rxv7"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.021051    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.521034143 +0000 UTC m=+1.842570939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021053    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-kubernetes\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021073    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-multus\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021106    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-conf-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021120    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c2280c9-7198-40c7-b6ac-fd299946a0d5-etc-sysctl-conf\") pod \"tuned-tddjp\" (UID: \"5c2280c9-7198-40c7-b6ac-fd299946a0d5\") " pod="openshift-cluster-node-tuning-operator/tuned-tddjp"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021146    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-multus-conf-dir\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.021164    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d387007-75e5-4186-8039-b96c77d34f1f-host-var-lib-cni-multus\") pod \"multus-7dkm6\" (UID: \"0d387007-75e5-4186-8039-b96c77d34f1f\") " pod="openshift-multus/multus-7dkm6"
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.021206    4550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:19.022158 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.021253    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:19.521239927 +0000 UTC m=+1.842776713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : secret "metrics-daemon-secret" not found
Nov 03 00:05:19.115794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.115771    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:05:19.525621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.525593    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:19.525761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.525638    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:19.525761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.525673    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:19.525761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.525722    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:19.525761 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525739    4550 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 00:05:19.525890 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525777    4550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.525890 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525824    4550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:19.525890 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525839    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.525820773 +0000 UTC m=+2.847357560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:19.525890 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525859    4550 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:19.526002 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525861    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.525849665 +0000 UTC m=+2.847386458 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : secret "metrics-daemon-secret" not found
Nov 03 00:05:19.526002 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525980    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.525958754 +0000 UTC m=+2.847495546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : configmap "kube-rbac-proxy" not found
Nov 03 00:05:19.526002 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:19.525993    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls podName:d5b7233c-5ec0-47fb-9129-793e2383b1bc nodeName:}" failed. No retries permitted until 2026-11-03 00:05:20.525987097 +0000 UTC m=+2.847523875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4cnb6" (UID: "d5b7233c-5ec0-47fb-9129-793e2383b1bc") : secret "insights-runtime-extractor-tls" not found
Nov 03 00:05:19.714511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.714481    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:19.765704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.765676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:19.842851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:19.842815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-drrmw" event={"ID":"5f6b2892-309f-49f9-94e2-9af7a736d740","Type":"ContainerStarted","Data":"db3354bd54beaf43a2f3ef8b10a1091b49004691db4c8558c44c8cfa8dda2a19"}
Nov 03 00:05:20.425327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.425289    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:20.525726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.525692    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:20.535874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.535842    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:20.536045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.535892    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:20.536045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.535924    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:20.536045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.535955    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:20.536209 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:20.536066    4550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:20.536209 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:20.536132    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.536114916 +0000 UTC m=+4.857651697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : secret "metrics-daemon-secret" not found
Nov 03 00:05:20.536499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.536477    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5b7233c-5ec0-47fb-9129-793e2383b1bc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:20.536557 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:20.536535    4550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:20.536609 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:20.536604    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:22.536565005 +0000 UTC m=+4.858101787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:20.536817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.536797    4550 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:05:20.541809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.539343    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5b7233c-5ec0-47fb-9129-793e2383b1bc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4cnb6\" (UID: \"d5b7233c-5ec0-47fb-9129-793e2383b1bc\") " pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:20.614153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.614109    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4cnb6"
Nov 03 00:05:20.628289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.628234    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:20.776942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.776911    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:20.877667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.877635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:20.934451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.934413    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4cnb6"]
Nov 03 00:05:20.937480 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:20.937444    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b7233c_5ec0_47fb_9129_793e2383b1bc.slice/crio-48cad1bba7b5c518e1a05effbf7e765c0a2ef6a315c2213f6c32df3c3d777555 WatchSource:0}: Error finding container 48cad1bba7b5c518e1a05effbf7e765c0a2ef6a315c2213f6c32df3c3d777555: Status 404 returned error can't find the container with id 48cad1bba7b5c518e1a05effbf7e765c0a2ef6a315c2213f6c32df3c3d777555
Nov 03 00:05:20.947709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:20.947693    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jkfg_a91bc30d-2c15-48ab-97c9-c54713750b36/dns-node-resolver/0.log"
Nov 03 00:05:21.570426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:21.570396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:21.854621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:21.854533    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cnb6" event={"ID":"d5b7233c-5ec0-47fb-9129-793e2383b1bc","Type":"ContainerStarted","Data":"b044f0b49c2b90cf862436897ba2b12a93ac671fb2343698b04637e1e612a845"}
Nov 03 00:05:21.854621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:21.854606    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cnb6" event={"ID":"d5b7233c-5ec0-47fb-9129-793e2383b1bc","Type":"ContainerStarted","Data":"c955c96a15bf96c28f13741dde843ed8a7f4bb6a41d344655256bc173f5b1853"}
Nov 03 00:05:21.854999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:21.854625    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cnb6" event={"ID":"d5b7233c-5ec0-47fb-9129-793e2383b1bc","Type":"ContainerStarted","Data":"48cad1bba7b5c518e1a05effbf7e765c0a2ef6a315c2213f6c32df3c3d777555"}
Nov 03 00:05:22.068674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.068642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:22.147737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.147662    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96xx2_887b3093-6795-4b74-aa85-f46f3dbc76fd/node-ca/0.log"
Nov 03 00:05:22.268652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.268602    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:22.469545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.469460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:22.549682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.549643    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:22.549877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.549710    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:22.549877 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:22.549773    4550 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 00:05:22.549877 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:22.549828    4550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 00:05:22.549877 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:22.549838    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls podName:24c625e0-13a7-48df-bd0e-c383ff56ecca nodeName:}" failed. No retries permitted until 2026-11-03 00:05:26.549819735 +0000 UTC m=+8.871356524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls") pod "dns-default-pzqlp" (UID: "24c625e0-13a7-48df-bd0e-c383ff56ecca") : secret "dns-default-metrics-tls" not found
Nov 03 00:05:22.550089 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:22.549897    4550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs podName:f9c9783e-067c-456e-b511-e687ca741b33 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:26.549883551 +0000 UTC m=+8.871420328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs") pod "network-metrics-daemon-t4t4w" (UID: "f9c9783e-067c-456e-b511-e687ca741b33") : secret "metrics-daemon-secret" not found
Nov 03 00:05:22.663991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.663957    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:22.860348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:22.860258    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:23.147647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:23.147580    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjlsx_4d168306-103a-47b1-9b32-0c33b7825416/serve-healthcheck-canary/0.log"
Nov 03 00:05:23.440045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:23.440008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:23.542519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:23.542494    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:23.542723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:23.542710    4550 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:05:23.548389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:23.548364    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-dd54j_80f326a9-e6d5-45df-b03d-37d6bec25926/migrator/0.log"
Nov 03 00:05:23.862968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:23.862886    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4cnb6" event={"ID":"d5b7233c-5ec0-47fb-9129-793e2383b1bc","Type":"ContainerStarted","Data":"b58ca9424d8fe63523685d15a182044d2ff5a145c032498c68ad0c009b7c5fe9"}
Nov 03 00:05:24.074455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:24.074398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.297243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:24.297208    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.468272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:24.468236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.698172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:24.698134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.721345 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:24.721303    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:05:24.864969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:24.864937    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:24.965527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:24.965421    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:25.065797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:25.065767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:25.374682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:25.374642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.066277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.066232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.254733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.254699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.465133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.465100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.582351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.582311    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:26.582351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.582356    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:26.584810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.584783    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/24c625e0-13a7-48df-bd0e-c383ff56ecca-metrics-tls\") pod \"dns-default-pzqlp\" (UID: \"24c625e0-13a7-48df-bd0e-c383ff56ecca\") " pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:26.584930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.584839    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9c9783e-067c-456e-b511-e687ca741b33-metrics-certs\") pod \"network-metrics-daemon-t4t4w\" (UID: \"f9c9783e-067c-456e-b511-e687ca741b33\") " pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:26.611822 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.611780    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4t4w"
Nov 03 00:05:26.612853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.612834    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:26.659353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.659313    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.865411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.865330    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:26.910900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.910870    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4t4w"]
Nov 03 00:05:26.911907 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:26.911881    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c9783e_067c_456e_b511_e687ca741b33.slice/crio-c6edb6e86ccb89b55deb91fcbe3a996637003787966f95a5648c4a115b68d5a0 WatchSource:0}: Error finding container c6edb6e86ccb89b55deb91fcbe3a996637003787966f95a5648c4a115b68d5a0: Status 404 returned error can't find the container with id c6edb6e86ccb89b55deb91fcbe3a996637003787966f95a5648c4a115b68d5a0
Nov 03 00:05:26.921278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:26.921241    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c625e0_13a7_48df_bd0e_c383ff56ecca.slice/crio-02a27259df4450e660271a156f9f4136a4b6ee3857e0da158dba083d7605296a WatchSource:0}: Error finding container 02a27259df4450e660271a156f9f4136a4b6ee3857e0da158dba083d7605296a: Status 404 returned error can't find the container with id 02a27259df4450e660271a156f9f4136a4b6ee3857e0da158dba083d7605296a
Nov 03 00:05:26.924135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:26.924106    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pzqlp"]
Nov 03 00:05:27.633458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:27.632979    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:27.641781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:27.641736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:27.876820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:27.876789    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pzqlp" event={"ID":"24c625e0-13a7-48df-bd0e-c383ff56ecca","Type":"ContainerStarted","Data":"02a27259df4450e660271a156f9f4136a4b6ee3857e0da158dba083d7605296a"}
Nov 03 00:05:27.878115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:27.878089    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4t4w" event={"ID":"f9c9783e-067c-456e-b511-e687ca741b33","Type":"ContainerStarted","Data":"c6edb6e86ccb89b55deb91fcbe3a996637003787966f95a5648c4a115b68d5a0"}
Nov 03 00:05:28.060751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.060722    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:28.267453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.262605    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:28.488240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.488206    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:28.690475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.690410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:28.869235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.869149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:28.886227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.886192    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pzqlp" event={"ID":"24c625e0-13a7-48df-bd0e-c383ff56ecca","Type":"ContainerStarted","Data":"999ffee9b8fa82ae4515a6ba181a111362b0456bacef444f4e048bd983e49573"}
Nov 03 00:05:28.886416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.886288    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:28.886416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.886313    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pzqlp" event={"ID":"24c625e0-13a7-48df-bd0e-c383ff56ecca","Type":"ContainerStarted","Data":"9fd2a9254ccbcbe117f26854c2cab42b09b3c89c055be67e32b4d1b157cd100a"}
Nov 03 00:05:28.887934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.887907    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4t4w" event={"ID":"f9c9783e-067c-456e-b511-e687ca741b33","Type":"ContainerStarted","Data":"3b00f7da3d00435724dfff79d16cbb90bcf678546b556e459896d8d0c86cffaa"}
Nov 03 00:05:28.888013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:28.887960    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4t4w" event={"ID":"f9c9783e-067c-456e-b511-e687ca741b33","Type":"ContainerStarted","Data":"e7f3498c51ad80082ef26f9cf2b869f5e567da94db614a83c5629911e46e0c2d"}
Nov 03 00:05:30.034348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.034311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.138085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.138048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.235341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.235305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.335745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.335653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.436332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.436299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.536541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.536509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.661466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.661421    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:30.864453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:30.864398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:31.463173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:31.463142    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:31.676655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:31.676622    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:31.789881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:31.789789    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:31.892555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:31.892522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:32.685487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.685455    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4zf7k"
Nov 03 00:05:32.828558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.828528    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-tm57s"]
Nov 03 00:05:32.831576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.831557    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:32.855127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.855102    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 00:05:32.855323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.855309    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 00:05:32.856067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.856053    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-ctkd8\""
Nov 03 00:05:32.869647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.869624    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tm57s"]
Nov 03 00:05:32.904330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.904306    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"]
Nov 03 00:05:32.907282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.907264    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:32.908790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.908772    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57dc984b8f-b8bms"]
Nov 03 00:05:32.911835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.911815    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:32.928251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.928227    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"]
Nov 03 00:05:32.934015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.933984    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e1bf19b-8b56-488d-bf10-9e3170b23393-tmp\") pod \"downloads-6dd789c8b7-tm57s\" (UID: \"7e1bf19b-8b56-488d-bf10-9e3170b23393\") " pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:32.934139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.934043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrk6z\" (UniqueName: \"kubernetes.io/projected/7e1bf19b-8b56-488d-bf10-9e3170b23393-kube-api-access-wrk6z\") pod \"downloads-6dd789c8b7-tm57s\" (UID: \"7e1bf19b-8b56-488d-bf10-9e3170b23393\") " pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:32.941932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.941875    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 00:05:32.942552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.942534    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57dc984b8f-b8bms"]
Nov 03 00:05:32.955383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.955365    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 00:05:32.955693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.955668    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 00:05:32.955693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.955679    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-68sq2\""
Nov 03 00:05:32.955837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.955708    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7x9g\""
Nov 03 00:05:32.955837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.955729    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 00:05:32.955837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.955736    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 00:05:32.982553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:32.982529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:33.034952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.034924    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkn2h\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-kube-api-access-wkn2h\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.034957    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a185d8a1-09e2-464d-b73b-cc11e1b5a068-registry-certificates\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.034980    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9610df3-39ba-4b02-8daa-71ec7facb86f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tqj4d\" (UID: \"c9610df3-39ba-4b02-8daa-71ec7facb86f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:33.035152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035079    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a185d8a1-09e2-464d-b73b-cc11e1b5a068-ca-trust-extracted\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035102    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a185d8a1-09e2-464d-b73b-cc11e1b5a068-installation-pull-secrets\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035131    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a185d8a1-09e2-464d-b73b-cc11e1b5a068-image-registry-private-configuration\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035171    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-registry-tls\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035252    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-bound-sa-token\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035291    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a185d8a1-09e2-464d-b73b-cc11e1b5a068-trusted-ca\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.035406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035341    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e1bf19b-8b56-488d-bf10-9e3170b23393-tmp\") pod \"downloads-6dd789c8b7-tm57s\" (UID: \"7e1bf19b-8b56-488d-bf10-9e3170b23393\") " pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:33.035406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035375    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrk6z\" (UniqueName: \"kubernetes.io/projected/7e1bf19b-8b56-488d-bf10-9e3170b23393-kube-api-access-wrk6z\") pod \"downloads-6dd789c8b7-tm57s\" (UID: \"7e1bf19b-8b56-488d-bf10-9e3170b23393\") " pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:33.035650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.035635    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e1bf19b-8b56-488d-bf10-9e3170b23393-tmp\") pod \"downloads-6dd789c8b7-tm57s\" (UID: \"7e1bf19b-8b56-488d-bf10-9e3170b23393\") " pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:33.061065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.061037    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrk6z\" (UniqueName: \"kubernetes.io/projected/7e1bf19b-8b56-488d-bf10-9e3170b23393-kube-api-access-wrk6z\") pod \"downloads-6dd789c8b7-tm57s\" (UID: \"7e1bf19b-8b56-488d-bf10-9e3170b23393\") " pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:33.135724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135693    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-bound-sa-token\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.135903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135733    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a185d8a1-09e2-464d-b73b-cc11e1b5a068-trusted-ca\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.135903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135772    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkn2h\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-kube-api-access-wkn2h\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.135903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135825    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a185d8a1-09e2-464d-b73b-cc11e1b5a068-registry-certificates\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.135903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135859    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9610df3-39ba-4b02-8daa-71ec7facb86f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tqj4d\" (UID: \"c9610df3-39ba-4b02-8daa-71ec7facb86f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:33.135903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135890    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a185d8a1-09e2-464d-b73b-cc11e1b5a068-ca-trust-extracted\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.136153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135911    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a185d8a1-09e2-464d-b73b-cc11e1b5a068-installation-pull-secrets\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.136153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135954    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a185d8a1-09e2-464d-b73b-cc11e1b5a068-image-registry-private-configuration\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.136153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.135980    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-registry-tls\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.136718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.136625    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a185d8a1-09e2-464d-b73b-cc11e1b5a068-ca-trust-extracted\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.136843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.136802    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a185d8a1-09e2-464d-b73b-cc11e1b5a068-registry-certificates\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.136843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.136802    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a185d8a1-09e2-464d-b73b-cc11e1b5a068-trusted-ca\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.138951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.138921    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a185d8a1-09e2-464d-b73b-cc11e1b5a068-image-registry-private-configuration\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.139024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.138977    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9610df3-39ba-4b02-8daa-71ec7facb86f-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-tqj4d\" (UID: \"c9610df3-39ba-4b02-8daa-71ec7facb86f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:33.139069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.139032    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a185d8a1-09e2-464d-b73b-cc11e1b5a068-installation-pull-secrets\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.139069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.139036    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-registry-tls\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.141832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.141816    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:33.156597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.156567    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-bound-sa-token\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.156884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.156859    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkn2h\" (UniqueName: \"kubernetes.io/projected/a185d8a1-09e2-464d-b73b-cc11e1b5a068-kube-api-access-wkn2h\") pod \"image-registry-57dc984b8f-b8bms\" (UID: \"a185d8a1-09e2-464d-b73b-cc11e1b5a068\") " pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.199850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.199715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:33.221409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.221387    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:33.226305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.226288    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.453097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.453072    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tm57s"]
Nov 03 00:05:33.453773 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:33.453747    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1bf19b_8b56_488d_bf10_9e3170b23393.slice/crio-62eb4ad49999119cb022604c8e058878d2652ec78ca9e044442870a750feec58 WatchSource:0}: Error finding container 62eb4ad49999119cb022604c8e058878d2652ec78ca9e044442870a750feec58: Status 404 returned error can't find the container with id 62eb4ad49999119cb022604c8e058878d2652ec78ca9e044442870a750feec58
Nov 03 00:05:33.516770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.516615    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"]
Nov 03 00:05:33.517425 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:33.517398    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9610df3_39ba_4b02_8daa_71ec7facb86f.slice/crio-a2b13b95038878776e67d6b3e5d6a84dfeb3434ec0baaf6aefc8c08f8a0dc092 WatchSource:0}: Error finding container a2b13b95038878776e67d6b3e5d6a84dfeb3434ec0baaf6aefc8c08f8a0dc092: Status 404 returned error can't find the container with id a2b13b95038878776e67d6b3e5d6a84dfeb3434ec0baaf6aefc8c08f8a0dc092
Nov 03 00:05:33.519994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.519973    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57dc984b8f-b8bms"]
Nov 03 00:05:33.520198 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:33.520174    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda185d8a1_09e2_464d_b73b_cc11e1b5a068.slice/crio-9f785dec1ba555f8a90a8274d5acb282d3e6d9d961ab6a25c2f2b557665bd185 WatchSource:0}: Error finding container 9f785dec1ba555f8a90a8274d5acb282d3e6d9d961ab6a25c2f2b557665bd185: Status 404 returned error can't find the container with id 9f785dec1ba555f8a90a8274d5acb282d3e6d9d961ab6a25c2f2b557665bd185
Nov 03 00:05:33.904032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.903994    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms" event={"ID":"a185d8a1-09e2-464d-b73b-cc11e1b5a068","Type":"ContainerStarted","Data":"62fd0b0eb09aff93e52580478c9f1cb7b0cc21bc060381ff22a23fae4cac0fc0"}
Nov 03 00:05:33.904483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.904051    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms" event={"ID":"a185d8a1-09e2-464d-b73b-cc11e1b5a068","Type":"ContainerStarted","Data":"9f785dec1ba555f8a90a8274d5acb282d3e6d9d961ab6a25c2f2b557665bd185"}
Nov 03 00:05:33.904483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.904073    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:33.905257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.905234    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d" event={"ID":"c9610df3-39ba-4b02-8daa-71ec7facb86f","Type":"ContainerStarted","Data":"a2b13b95038878776e67d6b3e5d6a84dfeb3434ec0baaf6aefc8c08f8a0dc092"}
Nov 03 00:05:33.906287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.906267    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tm57s" event={"ID":"7e1bf19b-8b56-488d-bf10-9e3170b23393","Type":"ContainerStarted","Data":"62eb4ad49999119cb022604c8e058878d2652ec78ca9e044442870a750feec58"}
Nov 03 00:05:33.945914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:33.945859    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms" podStartSLOduration=1.945845346 podStartE2EDuration="1.945845346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:33.945019976 +0000 UTC m=+16.266556776" watchObservedRunningTime="2026-11-03 00:05:33.945845346 +0000 UTC m=+16.267382146"
Nov 03 00:05:34.181158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:34.181090    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tnptp"
Nov 03 00:05:34.913764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:34.913726    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d" event={"ID":"c9610df3-39ba-4b02-8daa-71ec7facb86f","Type":"ContainerStarted","Data":"8e3989f10018aeada714ca17a5cca77476dc39d80713096b8791b86f04a05348"}
Nov 03 00:05:34.914201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:34.913959    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:34.920013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:34.919994    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d"
Nov 03 00:05:34.990192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:34.990143    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-tqj4d" podStartSLOduration=1.944079827 podStartE2EDuration="2.990127792s" totalImagesPullingTime="1.046047965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:34.952288856 +0000 UTC m=+17.273825656" watchObservedRunningTime="2026-11-03 00:05:34.990127792 +0000 UTC m=+17.311664594"
Nov 03 00:05:35.468377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:35.468337    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:35.568573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:35.568533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:35.654399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:35.654312    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:38.313449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:38.313399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:38.894205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:38.894173    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pzqlp"
Nov 03 00:05:39.933851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:39.933816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.139572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.139538    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.260381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.260298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.280367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.280332    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-66psj"]
Nov 03 00:05:40.286171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.286101    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.311368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.311338    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 00:05:40.317263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.317212    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56fpg\""
Nov 03 00:05:40.317398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.317287    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 00:05:40.360339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.360302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.384051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.384017    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 00:05:40.384051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.384015    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 00:05:40.384292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.384122    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 00:05:40.384292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.384258    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 00:05:40.389328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389300    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-root\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389341    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389414    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-sys\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389471    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-wtmp\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25sn\" (UniqueName: \"kubernetes.io/projected/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-kube-api-access-t25sn\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389617    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-textfile\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389648    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-tls\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.389726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389698    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-accelerators-collector-config\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.390092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.389726    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-metrics-client-ca\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.462028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.461983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.492559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492521    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-accelerators-collector-config\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.492752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492585    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-metrics-client-ca\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.492752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492617    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-root\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.492752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492642    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.492752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492672    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-sys\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.492752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492698    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-wtmp\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.492752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492724    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25sn\" (UniqueName: \"kubernetes.io/projected/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-kube-api-access-t25sn\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492774    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-textfile\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.492805    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-tls\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.493172    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-accelerators-collector-config\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.493224    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-sys\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.493369    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-wtmp\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.493513    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-root\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.493630    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-textfile\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.493884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.493746    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-metrics-client-ca\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.496009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.495982    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.498251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.498218    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-node-exporter-tls\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.515505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.515417    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25sn\" (UniqueName: \"kubernetes.io/projected/2fa98c9b-3ffb-4806-8ba0-8350e96b7517-kube-api-access-t25sn\") pod \"node-exporter-66psj\" (UID: \"2fa98c9b-3ffb-4806-8ba0-8350e96b7517\") " pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.571742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.571705    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.607986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.607954    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-66psj"
Nov 03 00:05:40.716962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.716923    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:40.937765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:40.937730    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66psj" event={"ID":"2fa98c9b-3ffb-4806-8ba0-8350e96b7517","Type":"ContainerStarted","Data":"50491530e3268ca08040dd97d446f81bfec91d3d7a0ed49e8794bada8ce4ac13"}
Nov 03 00:05:41.408335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:41.408301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:41.555599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:41.555541    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:41.679833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:41.679333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:41.943853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:41.943763    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66psj" event={"ID":"2fa98c9b-3ffb-4806-8ba0-8350e96b7517","Type":"ContainerStarted","Data":"a5437ce06c74453be683a019f981c978d80d1f21287b4e14981af1f90a574638"}
Nov 03 00:05:42.948819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:42.948787    4550 generic.go:356] "Generic (PLEG): container finished" podID="2fa98c9b-3ffb-4806-8ba0-8350e96b7517" containerID="a5437ce06c74453be683a019f981c978d80d1f21287b4e14981af1f90a574638" exitCode=0
Nov 03 00:05:42.949263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:42.948838    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66psj" event={"ID":"2fa98c9b-3ffb-4806-8ba0-8350e96b7517","Type":"ContainerDied","Data":"a5437ce06c74453be683a019f981c978d80d1f21287b4e14981af1f90a574638"}
Nov 03 00:05:43.529941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:43.529905    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:43.630180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:43.630138    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:43.730524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:43.730487    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:43.954490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:43.954418    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66psj" event={"ID":"2fa98c9b-3ffb-4806-8ba0-8350e96b7517","Type":"ContainerStarted","Data":"9fb1a2db624beaf9734f0ec12dece799ca6a25e42db43b47600df209d1513c97"}
Nov 03 00:05:43.954943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:43.954559    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-66psj" event={"ID":"2fa98c9b-3ffb-4806-8ba0-8350e96b7517","Type":"ContainerStarted","Data":"6fe633b57471fa56e271ad93857c1c5ea4cd8e771a95fb73394acf968b04329b"}
Nov 03 00:05:44.022551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:44.022500    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-66psj" podStartSLOduration=2.7720133799999997 podStartE2EDuration="4.022481015s" totalImagesPullingTime="1.180712705s" totalInitContainerRuntime="69.75493ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:44.00905137 +0000 UTC m=+26.330588174" watchObservedRunningTime="2026-11-03 00:05:44.022481015 +0000 UTC m=+26.344017818"
Nov 03 00:05:45.017644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:45.017611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:45.130834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:45.130797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:45.217669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:45.217631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:46.776285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:46.776204    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:46.878078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:46.878046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:47.246680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.246573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:47.259060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.259031    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:05:47.288280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.288250    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:05:47.288419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.288391    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.304526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.304503    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 00:05:47.330963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.330892    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 00:05:47.330963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.330929    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 00:05:47.330963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.330947    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 00:05:47.331200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331002    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 00:05:47.331200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331173    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 00:05:47.331293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331210    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 00:05:47.331293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331247    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cn2aftnor2akt\""
Nov 03 00:05:47.331384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331298    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xcmdk\""
Nov 03 00:05:47.331549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331455    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 00:05:47.331693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.331549    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 00:05:47.347951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.347923    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx84d\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.347964    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.347989    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348113    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348152    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348188    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348290    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348316    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348346    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348391    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348454    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348488    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348523    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348560    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348602    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348645    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348677    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348733    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348776    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.348825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.348807    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.404830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.404795    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 00:05:47.404830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.404830    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 00:05:47.405068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.405012    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 00:05:47.405358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.405332    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 00:05:47.405358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.405378    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 00:05:47.406165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.406147    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 00:05:47.449560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449535    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449601    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449625    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449650    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449676    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449712    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449740    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449770    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449797    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449830    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449861    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449899    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449928    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.449967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449957    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx84d\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.449985    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.450009    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.450039    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.450066    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.450092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.450759    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.450838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.450796    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.453096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.453070    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.453345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.453325    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.453585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.453566    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.453681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.453646    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.454187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.453834    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.454187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.454068    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.454338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.454269    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.454338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.454275    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.455288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.454837    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.455288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.455231    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.455288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.455235    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.456708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.456682    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.456828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.456762    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.456893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.456867    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.456939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.456870    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.457419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.457402    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.457813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.457797    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.474712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.474687    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx84d\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d\") pod \"prometheus-k8s-0\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.519900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.519839    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:47.609051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.609020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:47.636020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.635993    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vlgrc"
Nov 03 00:05:47.695417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.695385    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:05:47.898483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:47.898424    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:48.002612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:48.002576    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:48.053002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:48.052970    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:05:48.053646 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:05:48.053616    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8f8889_f294_40e8_8468_efc3e8f3b446.slice/crio-2c1e138ad43c7891dee6df959697c80eb95c6222aa98337500a4d826dc8cc5e0 WatchSource:0}: Error finding container 2c1e138ad43c7891dee6df959697c80eb95c6222aa98337500a4d826dc8cc5e0: Status 404 returned error can't find the container with id 2c1e138ad43c7891dee6df959697c80eb95c6222aa98337500a4d826dc8cc5e0
Nov 03 00:05:48.973290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:48.973251    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"2c1e138ad43c7891dee6df959697c80eb95c6222aa98337500a4d826dc8cc5e0"}
Nov 03 00:05:49.977958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:49.977928    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9" exitCode=0
Nov 03 00:05:49.978265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:49.977997    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9"}
Nov 03 00:05:50.848712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:50.848616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:50.953868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:50.953812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:50.985494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:50.985452    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tm57s" event={"ID":"7e1bf19b-8b56-488d-bf10-9e3170b23393","Type":"ContainerStarted","Data":"c8f7e3c922bdfa04e58b0ca1832263cbb1481221ea89148ff18c5c49ec5ad906"}
Nov 03 00:05:51.991096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:51.991006    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:52.002044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:52.002014    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-tm57s"
Nov 03 00:05:52.044748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:52.044692    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-tm57s" podStartSLOduration=3.587163923 podStartE2EDuration="20.044674172s" totalImagesPullingTime="16.457510249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:51.028032059 +0000 UTC m=+33.349568876" watchObservedRunningTime="2026-11-03 00:05:52.044674172 +0000 UTC m=+34.366210971"
Nov 03 00:05:52.672858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:52.672821    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:52.778179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:52.778134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:54.732733 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:05:54.732679    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:05:54.920815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:54.920709    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57dc984b8f-b8bms"
Nov 03 00:05:55.009324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:55.009219    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445"}
Nov 03 00:05:55.009324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:55.009302    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed"}
Nov 03 00:05:55.118176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:55.118114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:55.220034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:55.219952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:55.327499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:55.327375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:56.259827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:56.259669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:56.370505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:56.370385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:57.022029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:57.021998    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060"}
Nov 03 00:05:57.746593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:57.746561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:57.853393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:57.853358    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:58.033710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:58.033633    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71"}
Nov 03 00:05:58.033863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:58.033715    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba"}
Nov 03 00:05:58.033863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:58.033756    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerStarted","Data":"e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da"}
Nov 03 00:05:58.122525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:05:58.122461    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.231924646 podStartE2EDuration="11.122441907s" totalImagesPullingTime="7.885489101s" totalInitContainerRuntime="5.02816ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:58.108662208 +0000 UTC m=+40.430199007" watchObservedRunningTime="2026-11-03 00:05:58.122441907 +0000 UTC m=+40.443978698"
Nov 03 00:06:01.786961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:01.786872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:01.888454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:01.888409    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:02.696029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:02.695988    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:06.165658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:06.165620    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:06.251897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:06.251869    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:07.387193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:07.387154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:07.506307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:07.506272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:09.560479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:09.560442    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:10.044068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:10.044029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:10.122818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:10.122787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:10.710170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:10.710131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:10.813219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:10.813183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:10.900191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:10.900157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:16.781115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:16.781078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:16.883124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:16.883087    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:24.721185 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:24.721149    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:06:31.778483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:31.778446    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:31.868869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:31.868836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:37.690760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:37.690726    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:37.737513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:37.737486    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:38.174964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:38.174930    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:41.176126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:41.176089    4550 generic.go:356] "Generic (PLEG): container finished" podID="821da046-5713-4974-ad87-8debd07cc8bf" containerID="b0a2825ffeb13cbf45a3335940e1fc10ef9663a0052950d3a3d476da1819d154" exitCode=0
Nov 03 00:06:41.176636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:41.176166    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8whn6-nj4vj" event={"ID":"821da046-5713-4974-ad87-8debd07cc8bf","Type":"ContainerDied","Data":"b0a2825ffeb13cbf45a3335940e1fc10ef9663a0052950d3a3d476da1819d154"}
Nov 03 00:06:42.304051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.304026    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:06:42.492806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.492720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") pod \"821da046-5713-4974-ad87-8debd07cc8bf\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") "
Nov 03 00:06:42.492806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.492772    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") pod \"821da046-5713-4974-ad87-8debd07cc8bf\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") "
Nov 03 00:06:42.492806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.492791    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") pod \"821da046-5713-4974-ad87-8debd07cc8bf\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") "
Nov 03 00:06:42.493029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.492839    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") pod \"821da046-5713-4974-ad87-8debd07cc8bf\" (UID: \"821da046-5713-4974-ad87-8debd07cc8bf\") "
Nov 03 00:06:42.493169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.493141    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path" pod "821da046-5713-4974-ad87-8debd07cc8bf" (UID: "821da046-5713-4974-ad87-8debd07cc8bf"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:42.493216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.493170    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle" pod "821da046-5713-4974-ad87-8debd07cc8bf" (UID: "821da046-5713-4974-ad87-8debd07cc8bf"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:42.493216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.493174    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle" pod "821da046-5713-4974-ad87-8debd07cc8bf" (UID: "821da046-5713-4974-ad87-8debd07cc8bf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:42.495187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.495163    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86" pod "821da046-5713-4974-ad87-8debd07cc8bf" (UID: "821da046-5713-4974-ad87-8debd07cc8bf"). InnerVolumeSpecName "kube-api-access-smg86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:42.593686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.593651    4550 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/821da046-5713-4974-ad87-8debd07cc8bf-archives-path\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:42.593686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.593681    4550 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-service-ca-bundle\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:42.593686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.593694    4550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/821da046-5713-4974-ad87-8debd07cc8bf-trusted-ca-bundle\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:42.593917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:42.593704    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smg86\" (UniqueName: \"kubernetes.io/projected/821da046-5713-4974-ad87-8debd07cc8bf-kube-api-access-smg86\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:43.183444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:43.183400    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-8whn6-nj4vj"
Nov 03 00:06:43.183444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:43.183404    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-8whn6-nj4vj" event={"ID":"821da046-5713-4974-ad87-8debd07cc8bf","Type":"ContainerDied","Data":"df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65"}
Nov 03 00:06:43.183632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:43.183458    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df22a5c6877e95ce03eb914c6e406bf460d755ee05ee5ed3521f0a2c0e328d65"
Nov 03 00:06:43.688484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:43.688424    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:43.781335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:43.781302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:47.046928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:47.046891    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:47.166149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:47.166115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:54.722063 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:54.722026    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:06:56.719762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.719727    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:06:56.720380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.720341    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="prometheus" containerID="cri-o://c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed" gracePeriod=600
Nov 03 00:06:56.720546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.720375    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="thanos-sidecar" containerID="cri-o://d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060" gracePeriod=600
Nov 03 00:06:56.720546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.720370    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy" containerID="cri-o://45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba" gracePeriod=600
Nov 03 00:06:56.720546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.720419    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-web" containerID="cri-o://e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da" gracePeriod=600
Nov 03 00:06:56.720546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.720451    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="config-reloader" containerID="cri-o://e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445" gracePeriod=600
Nov 03 00:06:56.720546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:56.720468    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-thanos" containerID="cri-o://04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71" gracePeriod=600
Nov 03 00:06:57.236329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236298    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71" exitCode=0
Nov 03 00:06:57.236329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236322    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba" exitCode=0
Nov 03 00:06:57.236329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236329    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060" exitCode=0
Nov 03 00:06:57.236329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236335    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445" exitCode=0
Nov 03 00:06:57.236597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236340    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed" exitCode=0
Nov 03 00:06:57.236597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236368    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71"}
Nov 03 00:06:57.236597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236450    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba"}
Nov 03 00:06:57.236597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236481    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060"}
Nov 03 00:06:57.236597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236505    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445"}
Nov 03 00:06:57.236597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.236528    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed"}
Nov 03 00:06:57.955496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:57.955473    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.009223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009196    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009226    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009253    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009286    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009423    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009488    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009530    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009565    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009595    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009650    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009686    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009712    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009739    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.009766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009763    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009805    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009845    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009871    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009904    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009920    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.009933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.010009    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle\") pod \"fa8f8889-f294-40e8-8468-efc3e8f3b446\" (UID: \"fa8f8889-f294-40e8-8468-efc3e8f3b446\") "
Nov 03 00:06:58.010367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.010224    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.011079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.010840    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.011334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.011312    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:58.011588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.011558    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.012005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.011579    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.012509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.012241    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.012509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.012446    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.012654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.012628    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.013539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.012773    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "kube-api-access-kx84d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:58.013539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.012939    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:58.013539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.013162    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:58.014249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.014225    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.014410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.014382    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.014618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.014589    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:58.014713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.014693    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.014800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.014746    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.015012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.014982    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.015115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.015099    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.015377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.015359    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.024170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.024150    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config" pod "fa8f8889-f294-40e8-8468-efc3e8f3b446" (UID: "fa8f8889-f294-40e8-8468-efc3e8f3b446"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110813    4550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-tls-assets\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110838    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110850    4550 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110859    4550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110868    4550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-web-config\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110877    4550 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-kube-rbac-proxy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.110877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110885    4550 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-metrics-client-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110894    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110902    4550 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-config\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110910    4550 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-metrics-client-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110919    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110927    4550 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-grpc-tls\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110935    4550 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa8f8889-f294-40e8-8468-efc3e8f3b446-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110943    4550 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-prometheus-k8s-db\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110952    4550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110961    4550 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110970    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx84d\" (UniqueName: \"kubernetes.io/projected/fa8f8889-f294-40e8-8468-efc3e8f3b446-kube-api-access-kx84d\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110978    4550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa8f8889-f294-40e8-8468-efc3e8f3b446-config-out\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.111179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.110987    4550 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fa8f8889-f294-40e8-8468-efc3e8f3b446-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:06:58.242212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.242177    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerID="e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da" exitCode=0
Nov 03 00:06:58.242362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.242249    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da"}
Nov 03 00:06:58.242362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.242266    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.242362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.242318    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fa8f8889-f294-40e8-8468-efc3e8f3b446","Type":"ContainerDied","Data":"2c1e138ad43c7891dee6df959697c80eb95c6222aa98337500a4d826dc8cc5e0"}
Nov 03 00:06:58.242362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.242350    4550 scope.go:122] "RemoveContainer" containerID="04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71"
Nov 03 00:06:58.250378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.250324    4550 scope.go:122] "RemoveContainer" containerID="45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba"
Nov 03 00:06:58.257673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.257656    4550 scope.go:122] "RemoveContainer" containerID="e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da"
Nov 03 00:06:58.263863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.263845    4550 scope.go:122] "RemoveContainer" containerID="d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060"
Nov 03 00:06:58.269934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.269911    4550 scope.go:122] "RemoveContainer" containerID="e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445"
Nov 03 00:06:58.276120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.276104    4550 scope.go:122] "RemoveContainer" containerID="c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed"
Nov 03 00:06:58.282735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.282600    4550 scope.go:122] "RemoveContainer" containerID="e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9"
Nov 03 00:06:58.288632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.288614    4550 scope.go:122] "RemoveContainer" containerID="04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71"
Nov 03 00:06:58.288886 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.288868    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71\": container with ID starting with 04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71 not found: ID does not exist" containerID="04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71"
Nov 03 00:06:58.288936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.288892    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71"} err="failed to get container status \"04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71\": rpc error: code = NotFound desc = could not find container \"04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71\": container with ID starting with 04a2184a920fea1a570306b58bda9917f6b0c5665c409b63b41be3744d374e71 not found: ID does not exist"
Nov 03 00:06:58.288936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.288926    4550 scope.go:122] "RemoveContainer" containerID="45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba"
Nov 03 00:06:58.289159 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.289130    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba\": container with ID starting with 45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba not found: ID does not exist" containerID="45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba"
Nov 03 00:06:58.289230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289157    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba"} err="failed to get container status \"45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba\": rpc error: code = NotFound desc = could not find container \"45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba\": container with ID starting with 45833286f0273ab18996eccdc4fb1a46ebcdbc5f51b52af54aaab99ee329f4ba not found: ID does not exist"
Nov 03 00:06:58.289230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289178    4550 scope.go:122] "RemoveContainer" containerID="e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da"
Nov 03 00:06:58.289448 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.289416    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da\": container with ID starting with e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da not found: ID does not exist" containerID="e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da"
Nov 03 00:06:58.289499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289449    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da"} err="failed to get container status \"e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da\": rpc error: code = NotFound desc = could not find container \"e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da\": container with ID starting with e32d5561deaae590d657f7feae8c82987edd0f661f9366df822b214c2bfee6da not found: ID does not exist"
Nov 03 00:06:58.289499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289463    4550 scope.go:122] "RemoveContainer" containerID="d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060"
Nov 03 00:06:58.289653 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.289637    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060\": container with ID starting with d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060 not found: ID does not exist" containerID="d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060"
Nov 03 00:06:58.289690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289656    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060"} err="failed to get container status \"d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060\": rpc error: code = NotFound desc = could not find container \"d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060\": container with ID starting with d57887c7e46973d1e9513b4f07c3193e814c5e1c048b95cd7365e87a74519060 not found: ID does not exist"
Nov 03 00:06:58.289690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289670    4550 scope.go:122] "RemoveContainer" containerID="e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445"
Nov 03 00:06:58.289848 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.289835    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445\": container with ID starting with e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445 not found: ID does not exist" containerID="e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445"
Nov 03 00:06:58.289891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289854    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445"} err="failed to get container status \"e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445\": rpc error: code = NotFound desc = could not find container \"e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445\": container with ID starting with e551d261f06fdd359a407120b635a7f5502f69b2f6cde66afa30a765eff09445 not found: ID does not exist"
Nov 03 00:06:58.289891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.289866    4550 scope.go:122] "RemoveContainer" containerID="c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed"
Nov 03 00:06:58.290095 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.290080    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed\": container with ID starting with c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed not found: ID does not exist" containerID="c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed"
Nov 03 00:06:58.290137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.290097    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed"} err="failed to get container status \"c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed\": rpc error: code = NotFound desc = could not find container \"c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed\": container with ID starting with c98ff551f5bf38fca2241f7a2b3a3348b04d719f7315a1414839b0cd14af6aed not found: ID does not exist"
Nov 03 00:06:58.290137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.290108    4550 scope.go:122] "RemoveContainer" containerID="e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9"
Nov 03 00:06:58.290319 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:06:58.290304    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9\": container with ID starting with e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9 not found: ID does not exist" containerID="e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9"
Nov 03 00:06:58.290358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.290321    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9"} err="failed to get container status \"e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9\": rpc error: code = NotFound desc = could not find container \"e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9\": container with ID starting with e9548f007b91cb35b8d7e96a68f7cac892b43d3c7304b667c323062625bfc3d9 not found: ID does not exist"
Nov 03 00:06:58.295509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.295492    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:06:58.308442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.308412    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:06:58.319859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.319840    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:06:58.320058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320047    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy"
Nov 03 00:06:58.320095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320059    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy"
Nov 03 00:06:58.320095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320070    4550 cpu_manager.go:414] "containerMap: removing container" podUID="821da046-5713-4974-ad87-8debd07cc8bf" containerName="insights-gathering"
Nov 03 00:06:58.320095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320076    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821da046-5713-4974-ad87-8debd07cc8bf" containerName="insights-gathering"
Nov 03 00:06:58.320095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320089    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="config-reloader"
Nov 03 00:06:58.320095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320095    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="config-reloader"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320102    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-web"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320107    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-web"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320118    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="prometheus"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320123    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="prometheus"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320129    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="thanos-sidecar"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320134    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="thanos-sidecar"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320143    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="init-config-reloader"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320148    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="init-config-reloader"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320153    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-thanos"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320157    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-thanos"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320196    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="thanos-sidecar"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320203    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320210    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-thanos"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320215    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="821da046-5713-4974-ad87-8debd07cc8bf" containerName="insights-gathering"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320222    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="config-reloader"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320227    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="kube-rbac-proxy-web"
Nov 03 00:06:58.320239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.320232    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" containerName="prometheus"
Nov 03 00:06:58.325157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.325141    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.341183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341141    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 00:06:58.341278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341250    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 00:06:58.341345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341318    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 00:06:58.341491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341371    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 00:06:58.341491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341401    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 00:06:58.341491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341401    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 00:06:58.341876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341858    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 00:06:58.341985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341878    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 00:06:58.341985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.341948    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 00:06:58.342066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.342009    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 00:06:58.342066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.342024    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xcmdk\""
Nov 03 00:06:58.342066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.342028    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 00:06:58.342066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.342037    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-cn2aftnor2akt\""
Nov 03 00:06:58.342184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.342065    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 00:06:58.342184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.342085    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 00:06:58.355334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.355310    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 00:06:58.355487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.355470    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 00:06:58.356360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.356338    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:06:58.413867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.413846    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.413945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.413874    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.413945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.413899    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.413945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.413919    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.413972    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414044    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414085    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414103    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414143    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414174    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414201    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414246    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-web-config\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414264    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414296    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5x5\" (UniqueName: \"kubernetes.io/projected/661ba1e5-59ed-468f-a9ce-41483779bfcb-kube-api-access-pl5x5\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414334    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-config\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414351    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414396    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414453    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/661ba1e5-59ed-468f-a9ce-41483779bfcb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414475    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.414558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.414538    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/661ba1e5-59ed-468f-a9ce-41483779bfcb-config-out\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515122    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515149    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515167    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515191    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515320    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515360    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515393    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515418    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515474    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-web-config\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515500    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515526    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5x5\" (UniqueName: \"kubernetes.io/projected/661ba1e5-59ed-468f-a9ce-41483779bfcb-kube-api-access-pl5x5\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515552    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-config\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515576    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515629    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/661ba1e5-59ed-468f-a9ce-41483779bfcb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515655    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515684    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/661ba1e5-59ed-468f-a9ce-41483779bfcb-config-out\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515745    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.515790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515775    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.516086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515798    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.516086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515899    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.516086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.515929    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.516086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.516011    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.516633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.516584    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.517384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.517081    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518215    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518256    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518355    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518579    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-web-config\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518591    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518754    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.518884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.518865    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.519303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.519275    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.519644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.519612    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/661ba1e5-59ed-468f-a9ce-41483779bfcb-config-out\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.519959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.519939    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.520376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.520357    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.520574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.520556    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/661ba1e5-59ed-468f-a9ce-41483779bfcb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.521219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.521200    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/661ba1e5-59ed-468f-a9ce-41483779bfcb-config\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.521633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.521615    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/661ba1e5-59ed-468f-a9ce-41483779bfcb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.538326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.538306    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5x5\" (UniqueName: \"kubernetes.io/projected/661ba1e5-59ed-468f-a9ce-41483779bfcb-kube-api-access-pl5x5\") pod \"prometheus-k8s-0\" (UID: \"661ba1e5-59ed-468f-a9ce-41483779bfcb\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.648947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.648925    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:06:58.959616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:58.959589    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 00:06:58.960000 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:06:58.959965    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661ba1e5_59ed_468f_a9ce_41483779bfcb.slice/crio-19c8de02be875f164a9ac4ee01449ceb252690cf617fffc4fde3b1f499ca01b1 WatchSource:0}: Error finding container 19c8de02be875f164a9ac4ee01449ceb252690cf617fffc4fde3b1f499ca01b1: Status 404 returned error can't find the container with id 19c8de02be875f164a9ac4ee01449ceb252690cf617fffc4fde3b1f499ca01b1
Nov 03 00:06:59.247521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:59.247423    4550 generic.go:356] "Generic (PLEG): container finished" podID="661ba1e5-59ed-468f-a9ce-41483779bfcb" containerID="b6f2b4a9a2ca03b285d35d2055715065de62a5aaa45ba8b0d352b5b589686b8c" exitCode=0
Nov 03 00:06:59.247521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:59.247464    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerDied","Data":"b6f2b4a9a2ca03b285d35d2055715065de62a5aaa45ba8b0d352b5b589686b8c"}
Nov 03 00:06:59.247714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:59.247543    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"19c8de02be875f164a9ac4ee01449ceb252690cf617fffc4fde3b1f499ca01b1"}
Nov 03 00:06:59.647811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:59.647782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:59.727713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:59.727681    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:06:59.804872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:06:59.804844    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8f8889-f294-40e8-8468-efc3e8f3b446" path="/var/lib/kubelet/pods/fa8f8889-f294-40e8-8468-efc3e8f3b446/volumes"
Nov 03 00:07:00.254339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.254308    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"eb20893e9e166d35f5f278c2c34919e1e91329da2a35736895aefbfe14b0b8a3"}
Nov 03 00:07:00.254735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.254380    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"8f22f9317808d02a0ab3a138b6f28268ed2eb0740b2eea5a5f937dbb5dcf8733"}
Nov 03 00:07:00.254735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.254421    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"6e66dde5ed023ca76363f85c85e828b8cff50cd0efee7cc9d59e28b7dbbbd3a4"}
Nov 03 00:07:00.254735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.254476    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"582ac11110d675b51633012d37df036013ffa4e15a00036cff99022d8781dd85"}
Nov 03 00:07:00.254735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.254504    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"68f7bdb47438ade540a56ee5f30bd6b84485be1141133dbc9502accf9bb81c30"}
Nov 03 00:07:00.254735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.254527    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"661ba1e5-59ed-468f-a9ce-41483779bfcb","Type":"ContainerStarted","Data":"c04b5ab830aaa3c257ffba6c78c3f3914383c660bbe68645ac84d09c347e1793"}
Nov 03 00:07:00.317558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:00.317507    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.302035309 podStartE2EDuration="2.317493904s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.458595ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:07:00.31676653 +0000 UTC m=+102.638303379" watchObservedRunningTime="2026-11-03 00:07:00.317493904 +0000 UTC m=+102.639030704"
Nov 03 00:07:03.649667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:03.649635    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:07:07.481554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:07.481521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:10.164977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:10.164945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:10.463618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:10.463584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:16.784883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:16.784843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:16.885114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:16.885081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:16.960696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:16.960663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:07:24.722798 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:07:24.722760    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:07:54.721392 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:07:54.721314    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:07:58.649129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:58.649083    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:07:58.664715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:58.664691    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:07:59.474656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:07:59.474631    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 00:08:07.628264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:07.628231    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:24.721955 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:08:24.721916    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:08:48.302525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:48.302484    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:49.922855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:49.922810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:50.251815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:50.251737    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:50.361395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:50.361357    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:51.475989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:51.475958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:54.722446 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:08:54.722400    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:08:55.542645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:55.542599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:55.716631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:55.716594    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:56.418659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:56.418625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:56.574873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:56.574843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:56.673465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:56.673374    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:08:59.761692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:08:59.761650    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:07.777450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:07.777390    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:24.721465 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:09:24.721415    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:09:28.751714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:28.751676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:28.860530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:28.860493    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:34.321426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:34.321394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:49.481106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:49.481072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:52.723461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:52.723404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:53.285831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:53.285796    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:54.721288 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:09:54.721246    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:09:54.897888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:54.897857    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:55.034515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:55.034400    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:56.085653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.085619    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:56.240915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.240877    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"]
Nov 03 00:09:56.244081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.244059    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.281842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.281817    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"]
Nov 03 00:09:56.312890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.312864    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.313020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.312898    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.313020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.312928    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.313020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.313000    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.313159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.313044    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zv78\" (UniqueName: \"kubernetes.io/projected/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-kube-api-access-7zv78\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.313159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.313066    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.333991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.333948    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 00:09:56.333991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.333970    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 00:09:56.334192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.334175    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 00:09:56.334250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.334196    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kvgpd\""
Nov 03 00:09:56.334250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.334205    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 00:09:56.334250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.334207    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 00:09:56.392569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.392506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:56.413356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413335    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.413476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413375    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.413476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413392    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.413476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413415    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zv78\" (UniqueName: \"kubernetes.io/projected/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-kube-api-access-7zv78\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.413476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413459    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.413674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413484    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.413903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.413870    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.415931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.415909    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.416013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.415906    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.416054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.416022    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.416131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.416095    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.433730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.433709    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zv78\" (UniqueName: \"kubernetes.io/projected/7672d356-55b2-4ab1-b9aa-58ea949b5ea8-kube-api-access-7zv78\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-2jxm6\" (UID: \"7672d356-55b2-4ab1-b9aa-58ea949b5ea8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.500361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.500330    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:56.558353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.558330    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:09:56.610535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.609838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:56.795639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.795605    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:56.896619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.896578    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"]
Nov 03 00:09:56.896799 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:09:56.896777    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7672d356_55b2_4ab1_b9aa_58ea949b5ea8.slice/crio-0697a0b89215a26b11dee8f3519d1d24f879b7a51a98f7987a2963546c0681a0 WatchSource:0}: Error finding container 0697a0b89215a26b11dee8f3519d1d24f879b7a51a98f7987a2963546c0681a0: Status 404 returned error can't find the container with id 0697a0b89215a26b11dee8f3519d1d24f879b7a51a98f7987a2963546c0681a0
Nov 03 00:09:56.900370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:56.900349    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:57.004995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:57.004950    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:57.846026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:57.845955    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6" event={"ID":"7672d356-55b2-4ab1-b9aa-58ea949b5ea8","Type":"ContainerStarted","Data":"0697a0b89215a26b11dee8f3519d1d24f879b7a51a98f7987a2963546c0681a0"}
Nov 03 00:09:57.888171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:57.888132    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:09:59.854830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:59.854801    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6" event={"ID":"7672d356-55b2-4ab1-b9aa-58ea949b5ea8","Type":"ContainerStarted","Data":"0b28c0e28e0786f6ef28468bbc742f067fa5f5b27e717d4c536dbfb22dc24b19"}
Nov 03 00:09:59.937302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:09:59.937266    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:00.480613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:00.480577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:00.859556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:00.859524    4550 generic.go:356] "Generic (PLEG): container finished" podID="7672d356-55b2-4ab1-b9aa-58ea949b5ea8" containerID="0b28c0e28e0786f6ef28468bbc742f067fa5f5b27e717d4c536dbfb22dc24b19" exitCode=0
Nov 03 00:10:00.859945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:00.859620    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6" event={"ID":"7672d356-55b2-4ab1-b9aa-58ea949b5ea8","Type":"ContainerDied","Data":"0b28c0e28e0786f6ef28468bbc742f067fa5f5b27e717d4c536dbfb22dc24b19"}
Nov 03 00:10:01.275999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:01.275964    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:01.864990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:01.864957    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6" event={"ID":"7672d356-55b2-4ab1-b9aa-58ea949b5ea8","Type":"ContainerStarted","Data":"3be47d3a5d1a0f89ef5d945e30ca2d9ea1f4cefaae3dd7badf849fcf50b5ad7c"}
Nov 03 00:10:01.865360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:01.865104    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:10:01.951082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:01.951033    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6" podStartSLOduration=2.2706924600000002 podStartE2EDuration="5.951018845s" totalImagesPullingTime="3.239761344s" totalInitContainerRuntime="440.565041ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:01.937107929 +0000 UTC m=+284.258644730" watchObservedRunningTime="2026-11-03 00:10:01.951018845 +0000 UTC m=+284.272555644"
Nov 03 00:10:02.082112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:02.082079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:02.660714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:02.660678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:02.816804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:02.816766    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:05.185535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:05.185496    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:05.512499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:05.512405    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:06.117023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.116991    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"]
Nov 03 00:10:06.123440 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.123417    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.153919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.153892    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 00:10:06.154065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.154048    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:10:06.154130 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.154079    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-xwrrp\""
Nov 03 00:10:06.155050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.155030    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"]
Nov 03 00:10:06.192505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.192480    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c41fbe1-529a-4a69-83f5-1b2ba7a87320-tmp\") pod \"jobset-operator-674cd59dc5-gls75\" (UID: \"4c41fbe1-529a-4a69-83f5-1b2ba7a87320\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.192824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.192514    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqsmg\" (UniqueName: \"kubernetes.io/projected/4c41fbe1-529a-4a69-83f5-1b2ba7a87320-kube-api-access-tqsmg\") pod \"jobset-operator-674cd59dc5-gls75\" (UID: \"4c41fbe1-529a-4a69-83f5-1b2ba7a87320\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.293051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.293022    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c41fbe1-529a-4a69-83f5-1b2ba7a87320-tmp\") pod \"jobset-operator-674cd59dc5-gls75\" (UID: \"4c41fbe1-529a-4a69-83f5-1b2ba7a87320\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.293182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.293071    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqsmg\" (UniqueName: \"kubernetes.io/projected/4c41fbe1-529a-4a69-83f5-1b2ba7a87320-kube-api-access-tqsmg\") pod \"jobset-operator-674cd59dc5-gls75\" (UID: \"4c41fbe1-529a-4a69-83f5-1b2ba7a87320\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.293398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.293378    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c41fbe1-529a-4a69-83f5-1b2ba7a87320-tmp\") pod \"jobset-operator-674cd59dc5-gls75\" (UID: \"4c41fbe1-529a-4a69-83f5-1b2ba7a87320\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.314758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.314734    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqsmg\" (UniqueName: \"kubernetes.io/projected/4c41fbe1-529a-4a69-83f5-1b2ba7a87320-kube-api-access-tqsmg\") pod \"jobset-operator-674cd59dc5-gls75\" (UID: \"4c41fbe1-529a-4a69-83f5-1b2ba7a87320\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.449900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.449866    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"
Nov 03 00:10:06.741286 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:06.741259    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c41fbe1_529a_4a69_83f5_1b2ba7a87320.slice/crio-a4d0c53b703be20e49d62df3cdf7299fab9fde28ab9e52fd3200a8059c7f47e2 WatchSource:0}: Error finding container a4d0c53b703be20e49d62df3cdf7299fab9fde28ab9e52fd3200a8059c7f47e2: Status 404 returned error can't find the container with id a4d0c53b703be20e49d62df3cdf7299fab9fde28ab9e52fd3200a8059c7f47e2
Nov 03 00:10:06.741518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.741494    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gls75"]
Nov 03 00:10:06.882557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:06.882520    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75" event={"ID":"4c41fbe1-529a-4a69-83f5-1b2ba7a87320","Type":"ContainerStarted","Data":"a4d0c53b703be20e49d62df3cdf7299fab9fde28ab9e52fd3200a8059c7f47e2"}
Nov 03 00:10:07.847373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:07.847292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:08.124452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.124347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:08.469189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.469159    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:08.909559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.908406    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"]
Nov 03 00:10:08.914985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.914963    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:08.924319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.924276    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"]
Nov 03 00:10:08.936346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.936314    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 00:10:08.936346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.936313    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l2kb6\""
Nov 03 00:10:08.936520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:08.936342    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 00:10:09.018676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.018647    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gfl\" (UniqueName: \"kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m82cj\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.018827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.018696    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m82cj\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.120135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.120099    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gfl\" (UniqueName: \"kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m82cj\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.120314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.120165    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m82cj\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.120618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.120596    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m82cj\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.140951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.140923    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gfl\" (UniqueName: \"kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl\") pod \"kuadrant-operator-controller-manager-cf8469d7b-m82cj\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.232597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.232514    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:09.580088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.580062    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"]
Nov 03 00:10:09.581132 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:09.581109    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a283e6b_0101_4688_b96c_41b549f9f753.slice/crio-18a6a2cecd089268a9fa63cd5a0b2be0bd6234813bb55e4c3fbc8953f9414f48 WatchSource:0}: Error finding container 18a6a2cecd089268a9fa63cd5a0b2be0bd6234813bb55e4c3fbc8953f9414f48: Status 404 returned error can't find the container with id 18a6a2cecd089268a9fa63cd5a0b2be0bd6234813bb55e4c3fbc8953f9414f48
Nov 03 00:10:09.898347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.898270    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj" event={"ID":"3a283e6b-0101-4688-b96c-41b549f9f753","Type":"ContainerStarted","Data":"18a6a2cecd089268a9fa63cd5a0b2be0bd6234813bb55e4c3fbc8953f9414f48"}
Nov 03 00:10:09.899985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.899953    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75" event={"ID":"4c41fbe1-529a-4a69-83f5-1b2ba7a87320","Type":"ContainerStarted","Data":"31c579a6cb41a90fe025372ba035dcd4e4c0eec54045c223fd69abcce2e1a4b9"}
Nov 03 00:10:09.960889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.960844    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gls75" podStartSLOduration=1.152255568 podStartE2EDuration="3.960828348s" totalImagesPullingTime="2.80857278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:09.960069469 +0000 UTC m=+292.281606269" watchObservedRunningTime="2026-11-03 00:10:09.960828348 +0000 UTC m=+292.282365147"
Nov 03 00:10:09.974637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:09.974611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:10.331110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:10.331072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:12.871114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:12.871084    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-2jxm6"
Nov 03 00:10:12.909097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:12.909062    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:13.024667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:13.024634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:17.603282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:17.603248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:17.740214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:17.740182    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:17.933202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:17.933164    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj" event={"ID":"3a283e6b-0101-4688-b96c-41b549f9f753","Type":"ContainerStarted","Data":"e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570"}
Nov 03 00:10:17.933382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:17.933271    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:18.038418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:18.038354    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj" podStartSLOduration=2.739521787 podStartE2EDuration="10.038335315s" totalImagesPullingTime="7.298813528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:18.024846699 +0000 UTC m=+300.346383517" watchObservedRunningTime="2026-11-03 00:10:18.038335315 +0000 UTC m=+300.359872115"
Nov 03 00:10:18.280168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:18.280077    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:18.405308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:18.405268    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:19.455623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.455590    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:19.555603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.555570    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:19.702933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.702901    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"]
Nov 03 00:10:19.704953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.704926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:19.707895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.707838    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.724836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.724810    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"]
Nov 03 00:10:19.724957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.724886    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 00:10:19.736125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.736102    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 00:10:19.736223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.736138    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cl6jh\""
Nov 03 00:10:19.812878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.812843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:19.813833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.813814    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/624eee0c-d8af-4382-876c-f8ddac59b3cf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.813900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.813865    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xh8\" (UniqueName: \"kubernetes.io/projected/624eee0c-d8af-4382-876c-f8ddac59b3cf-kube-api-access-m7xh8\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.813960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.813944    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/624eee0c-d8af-4382-876c-f8ddac59b3cf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.914897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.914864    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xh8\" (UniqueName: \"kubernetes.io/projected/624eee0c-d8af-4382-876c-f8ddac59b3cf-kube-api-access-m7xh8\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.915062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.914900    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/624eee0c-d8af-4382-876c-f8ddac59b3cf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.915062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.914950    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/624eee0c-d8af-4382-876c-f8ddac59b3cf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.915707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.915687    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/624eee0c-d8af-4382-876c-f8ddac59b3cf-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.917377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.917356    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/624eee0c-d8af-4382-876c-f8ddac59b3cf-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:19.939081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:19.939037    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xh8\" (UniqueName: \"kubernetes.io/projected/624eee0c-d8af-4382-876c-f8ddac59b3cf-kube-api-access-m7xh8\") pod \"kuadrant-console-plugin-66c4fb575d-jk4jg\" (UID: \"624eee0c-d8af-4382-876c-f8ddac59b3cf\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:20.020727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:20.020656    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"
Nov 03 00:10:20.086584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:20.086531    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:20.310491 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:20.310459    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod624eee0c_d8af_4382_876c_f8ddac59b3cf.slice/crio-267305c0646380d9eb2a609b628379f10c9b812851f3d183124aa5e177d5b5b8 WatchSource:0}: Error finding container 267305c0646380d9eb2a609b628379f10c9b812851f3d183124aa5e177d5b5b8: Status 404 returned error can't find the container with id 267305c0646380d9eb2a609b628379f10c9b812851f3d183124aa5e177d5b5b8
Nov 03 00:10:20.310899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:20.310874    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg"]
Nov 03 00:10:20.311789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:20.311772    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:10:20.948108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:20.948068    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg" event={"ID":"624eee0c-d8af-4382-876c-f8ddac59b3cf","Type":"ContainerStarted","Data":"267305c0646380d9eb2a609b628379f10c9b812851f3d183124aa5e177d5b5b8"}
Nov 03 00:10:23.424635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:23.424585    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:23.538061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:23.538021    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:23.685684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:23.685602    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:23.778225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:23.778185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:24.727627 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:10:24.727589    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:10:25.969206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:25.969173    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg" event={"ID":"624eee0c-d8af-4382-876c-f8ddac59b3cf","Type":"ContainerStarted","Data":"915b7e3498ab4d1b5268640aa1c937e7a3eb8aad227545e89e39e87f5cc1852e"}
Nov 03 00:10:26.010504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:26.010454    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-jk4jg" podStartSLOduration=2.074581173 podStartE2EDuration="7.010426694s" totalImagesPullingTime="4.935845521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:26.009160318 +0000 UTC m=+308.330697118" watchObservedRunningTime="2026-11-03 00:10:26.010426694 +0000 UTC m=+308.331963487"
Nov 03 00:10:28.940170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:28.940134    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:28.969912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:28.969878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:29.021040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.021008    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-4rsd4"]
Nov 03 00:10:29.024802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.024782    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.042281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.042248    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 00:10:29.042281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.042248    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 00:10:29.042410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.042392    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-62nhj\""
Nov 03 00:10:29.047899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.047859    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4rsd4"]
Nov 03 00:10:29.080823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.080794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:29.192207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.192145    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqfm6\" (UniqueName: \"kubernetes.io/projected/d8a6bce9-4c0f-4af0-985c-092bae88ef77-kube-api-access-bqfm6\") pod \"cert-manager-597b95cb84-4rsd4\" (UID: \"d8a6bce9-4c0f-4af0-985c-092bae88ef77\") " pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.192207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.192178    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8a6bce9-4c0f-4af0-985c-092bae88ef77-bound-sa-token\") pod \"cert-manager-597b95cb84-4rsd4\" (UID: \"d8a6bce9-4c0f-4af0-985c-092bae88ef77\") " pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.292591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.292556    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqfm6\" (UniqueName: \"kubernetes.io/projected/d8a6bce9-4c0f-4af0-985c-092bae88ef77-kube-api-access-bqfm6\") pod \"cert-manager-597b95cb84-4rsd4\" (UID: \"d8a6bce9-4c0f-4af0-985c-092bae88ef77\") " pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.292689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.292600    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8a6bce9-4c0f-4af0-985c-092bae88ef77-bound-sa-token\") pod \"cert-manager-597b95cb84-4rsd4\" (UID: \"d8a6bce9-4c0f-4af0-985c-092bae88ef77\") " pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.318841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.318821    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d8a6bce9-4c0f-4af0-985c-092bae88ef77-bound-sa-token\") pod \"cert-manager-597b95cb84-4rsd4\" (UID: \"d8a6bce9-4c0f-4af0-985c-092bae88ef77\") " pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.319148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.319127    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqfm6\" (UniqueName: \"kubernetes.io/projected/d8a6bce9-4c0f-4af0-985c-092bae88ef77-kube-api-access-bqfm6\") pod \"cert-manager-597b95cb84-4rsd4\" (UID: \"d8a6bce9-4c0f-4af0-985c-092bae88ef77\") " pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.336781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.336762    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-4rsd4"
Nov 03 00:10:29.375688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.375659    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"]
Nov 03 00:10:29.379913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.379897    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.412046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.412018    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"]
Nov 03 00:10:29.494770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.494674    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwxj\" (UniqueName: \"kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj\") pod \"kuadrant-operator-controller-manager-678d87766f-89stt\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.494770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.494718    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-89stt\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.595606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.595578    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwxj\" (UniqueName: \"kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj\") pod \"kuadrant-operator-controller-manager-678d87766f-89stt\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.595742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.595616    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-89stt\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.596039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.596022    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-89stt\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.616292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.616260    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwxj\" (UniqueName: \"kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj\") pod \"kuadrant-operator-controller-manager-678d87766f-89stt\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.626195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.626169    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-4rsd4"]
Nov 03 00:10:29.626399 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:29.626378    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8a6bce9_4c0f_4af0_985c_092bae88ef77.slice/crio-cbfea1fa5840d2702869eb58ad5251f1113c98c610ffa9058c34c6d25d02d94b WatchSource:0}: Error finding container cbfea1fa5840d2702869eb58ad5251f1113c98c610ffa9058c34c6d25d02d94b: Status 404 returned error can't find the container with id cbfea1fa5840d2702869eb58ad5251f1113c98c610ffa9058c34c6d25d02d94b
Nov 03 00:10:29.695448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.695409    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:29.985314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:29.985284    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4rsd4" event={"ID":"d8a6bce9-4c0f-4af0-985c-092bae88ef77","Type":"ContainerStarted","Data":"cbfea1fa5840d2702869eb58ad5251f1113c98c610ffa9058c34c6d25d02d94b"}
Nov 03 00:10:30.013181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:30.013155    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"]
Nov 03 00:10:30.013529 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:30.013503    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1877da_8248_4ab3_b721_c154ffa7e09f.slice/crio-961b106cbef5a60c03551c9752a6695c20fa6bebee5b94a43485f4b4880ecfbd WatchSource:0}: Error finding container 961b106cbef5a60c03551c9752a6695c20fa6bebee5b94a43485f4b4880ecfbd: Status 404 returned error can't find the container with id 961b106cbef5a60c03551c9752a6695c20fa6bebee5b94a43485f4b4880ecfbd
Nov 03 00:10:30.062219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:30.062196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:30.993197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:30.993162    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt" event={"ID":"0b1877da-8248-4ab3-b721-c154ffa7e09f","Type":"ContainerStarted","Data":"cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17"}
Nov 03 00:10:30.993805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:30.993223    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt" event={"ID":"0b1877da-8248-4ab3-b721-c154ffa7e09f","Type":"ContainerStarted","Data":"961b106cbef5a60c03551c9752a6695c20fa6bebee5b94a43485f4b4880ecfbd"}
Nov 03 00:10:30.994040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:30.994015    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:31.063086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:31.063025    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt" podStartSLOduration=2.063010676 podStartE2EDuration="2.063010676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:31.048485175 +0000 UTC m=+313.370021997" watchObservedRunningTime="2026-11-03 00:10:31.063010676 +0000 UTC m=+313.384547475"
Nov 03 00:10:31.124543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:31.124498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:31.226722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:31.226691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:31.568697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:31.568543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:31.745045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:31.745011    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:32.070168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:32.070131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:32.261339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:32.261302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:32.397403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:32.397325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:32.748786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:32.748746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:32.922801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:32.922773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:33.003793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.003717    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-4rsd4" event={"ID":"d8a6bce9-4c0f-4af0-985c-092bae88ef77","Type":"ContainerStarted","Data":"9f8e1f3460b451a38d3681fab6e671b8702f4683a4626193c64da666ba7facac"}
Nov 03 00:10:33.048096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.048051    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-4rsd4" podStartSLOduration=1.760734638 podStartE2EDuration="5.048037506s" totalImagesPullingTime="3.287302868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:33.047586242 +0000 UTC m=+315.369123055" watchObservedRunningTime="2026-11-03 00:10:33.048037506 +0000 UTC m=+315.369574305"
Nov 03 00:10:33.336665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.336554    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:33.552707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.552672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:33.669597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.669568    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:33.672838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.672822    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.699348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.699319    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:33.710672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.710641    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 00:10:33.735578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.735540    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:33.810593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.810564    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:33.831790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.831755    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5r9\" (UniqueName: \"kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9\") pod \"limitador-limitador-74dfcfb894-xnl45\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.831957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.831808    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file\") pod \"limitador-limitador-74dfcfb894-xnl45\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.839667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.839636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:33.932827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.932735    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file\") pod \"limitador-limitador-74dfcfb894-xnl45\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.932827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.932800    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5r9\" (UniqueName: \"kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9\") pod \"limitador-limitador-74dfcfb894-xnl45\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.933371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.933351    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file\") pod \"limitador-limitador-74dfcfb894-xnl45\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.952162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.952126    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5r9\" (UniqueName: \"kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9\") pod \"limitador-limitador-74dfcfb894-xnl45\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:33.984804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:33.984773    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:34.044001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.043969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:34.282057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.282036    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:34.282537 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:34.282513    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6975b04f_4fb0_4680_9c99_1d522c39dd04.slice/crio-e574d387f5690a396761ee12f60d1bb64aa8fce53349aab4328b8846a4d4eff2 WatchSource:0}: Error finding container e574d387f5690a396761ee12f60d1bb64aa8fce53349aab4328b8846a4d4eff2: Status 404 returned error can't find the container with id e574d387f5690a396761ee12f60d1bb64aa8fce53349aab4328b8846a4d4eff2
Nov 03 00:10:34.544874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.544796    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r2b47"]
Nov 03 00:10:34.549360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.549340    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:34.565955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.565925    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqpgv\""
Nov 03 00:10:34.571189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.571163    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r2b47"]
Nov 03 00:10:34.625760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.625720    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:34.639851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.639822    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9l5n\" (UniqueName: \"kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n\") pod \"authorino-57f7f8659f-r2b47\" (UID: \"73bd39cb-cdba-435b-921d-94e4da321308\") " pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:34.741047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.741015    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9l5n\" (UniqueName: \"kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n\") pod \"authorino-57f7f8659f-r2b47\" (UID: \"73bd39cb-cdba-435b-921d-94e4da321308\") " pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:34.760980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.760944    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9l5n\" (UniqueName: \"kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n\") pod \"authorino-57f7f8659f-r2b47\" (UID: \"73bd39cb-cdba-435b-921d-94e4da321308\") " pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:34.803173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.803089    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:34.860188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:34.860163    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:35.016156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:35.016126    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45" event={"ID":"6975b04f-4fb0-4680-9c99-1d522c39dd04","Type":"ContainerStarted","Data":"e574d387f5690a396761ee12f60d1bb64aa8fce53349aab4328b8846a4d4eff2"}
Nov 03 00:10:35.146854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:35.146825    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r2b47"]
Nov 03 00:10:35.147040 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:35.147017    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73bd39cb_cdba_435b_921d_94e4da321308.slice/crio-4febf7e8fac8a57bb175596fbba0a131b55d7cdfdca07a790931daec0463ed85 WatchSource:0}: Error finding container 4febf7e8fac8a57bb175596fbba0a131b55d7cdfdca07a790931daec0463ed85: Status 404 returned error can't find the container with id 4febf7e8fac8a57bb175596fbba0a131b55d7cdfdca07a790931daec0463ed85
Nov 03 00:10:35.352372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:35.352317    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:36.021809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:36.021771    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45" event={"ID":"6975b04f-4fb0-4680-9c99-1d522c39dd04","Type":"ContainerStarted","Data":"b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3"}
Nov 03 00:10:36.022265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:36.021882    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:36.023023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:36.023002    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r2b47" event={"ID":"73bd39cb-cdba-435b-921d-94e4da321308","Type":"ContainerStarted","Data":"4febf7e8fac8a57bb175596fbba0a131b55d7cdfdca07a790931daec0463ed85"}
Nov 03 00:10:36.036939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:36.036908    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:36.063020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:36.062978    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45" podStartSLOduration=1.720015686 podStartE2EDuration="3.062965834s" totalImagesPullingTime="1.342950148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:36.061944159 +0000 UTC m=+318.383480979" watchObservedRunningTime="2026-11-03 00:10:36.062965834 +0000 UTC m=+318.384502631"
Nov 03 00:10:36.137197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:36.137158    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:10:37.722401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:37.722364    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r2b47"]
Nov 03 00:10:39.037083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.037043    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r2b47" event={"ID":"73bd39cb-cdba-435b-921d-94e4da321308","Type":"ContainerStarted","Data":"ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1"}
Nov 03 00:10:39.037555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.037096    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-r2b47" podUID="73bd39cb-cdba-435b-921d-94e4da321308" containerName="authorino" containerID="cri-o://ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1" gracePeriod=30
Nov 03 00:10:39.076494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.076455    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-r2b47" podStartSLOduration=2.091559246 podStartE2EDuration="5.076422486s" totalImagesPullingTime="2.98486324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:39.075939067 +0000 UTC m=+321.397475868" watchObservedRunningTime="2026-11-03 00:10:39.076422486 +0000 UTC m=+321.397959337"
Nov 03 00:10:39.272532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.272509    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:39.279079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.279063    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n\" (UniqueName: \"kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n\") pod \"73bd39cb-cdba-435b-921d-94e4da321308\" (UID: \"73bd39cb-cdba-435b-921d-94e4da321308\") "
Nov 03 00:10:39.281020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.280997    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n" pod "73bd39cb-cdba-435b-921d-94e4da321308" (UID: "73bd39cb-cdba-435b-921d-94e4da321308"). InnerVolumeSpecName "kube-api-access-q9l5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:10:39.379894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:39.379801    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9l5n\" (UniqueName: \"kubernetes.io/projected/73bd39cb-cdba-435b-921d-94e4da321308-kube-api-access-q9l5n\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:10:40.042328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.042284    4550 generic.go:356] "Generic (PLEG): container finished" podID="73bd39cb-cdba-435b-921d-94e4da321308" containerID="ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1" exitCode=0
Nov 03 00:10:40.042328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.042320    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r2b47"
Nov 03 00:10:40.042850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.042366    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r2b47" event={"ID":"73bd39cb-cdba-435b-921d-94e4da321308","Type":"ContainerDied","Data":"ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1"}
Nov 03 00:10:40.042850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.042416    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r2b47" event={"ID":"73bd39cb-cdba-435b-921d-94e4da321308","Type":"ContainerDied","Data":"4febf7e8fac8a57bb175596fbba0a131b55d7cdfdca07a790931daec0463ed85"}
Nov 03 00:10:40.042850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.042460    4550 scope.go:122] "RemoveContainer" containerID="ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1"
Nov 03 00:10:40.050178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.050162    4550 scope.go:122] "RemoveContainer" containerID="ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1"
Nov 03 00:10:40.050456 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:10:40.050420    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1\": container with ID starting with ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1 not found: ID does not exist" containerID="ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1"
Nov 03 00:10:40.050554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.050462    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1"} err="failed to get container status \"ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1\": rpc error: code = NotFound desc = could not find container \"ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1\": container with ID starting with ec419dc4ce6d703938d4da9d7f8824930d857a0bd97a5930a39afd13e6526af1 not found: ID does not exist"
Nov 03 00:10:40.093315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.093238    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r2b47"]
Nov 03 00:10:40.106243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:40.106218    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r2b47"]
Nov 03 00:10:41.309366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.309333    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"]
Nov 03 00:10:41.309737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.309672    4550 cpu_manager.go:414] "containerMap: removing container" podUID="73bd39cb-cdba-435b-921d-94e4da321308" containerName="authorino"
Nov 03 00:10:41.309737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.309685    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bd39cb-cdba-435b-921d-94e4da321308" containerName="authorino"
Nov 03 00:10:41.309815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.309756    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bd39cb-cdba-435b-921d-94e4da321308" containerName="authorino"
Nov 03 00:10:41.314441 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.314414    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.329527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.329491    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 00:10:41.342390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.342350    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 00:10:41.342638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.342400    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-j2jlk\""
Nov 03 00:10:41.368828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.368802    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 00:10:41.370124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.370106    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"]
Nov 03 00:10:41.396151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.396123    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.396236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.396172    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.396319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.396301    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnb4\" (UniqueName: \"kubernetes.io/projected/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-kube-api-access-bpnb4\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.396374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.396347    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.497582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.497550    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.497717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.497589    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.497717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.497629    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.497717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.497700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnb4\" (UniqueName: \"kubernetes.io/projected/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-kube-api-access-bpnb4\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.497930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.497912    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.498097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.498076    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.498520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.498500    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.517175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.517143    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnb4\" (UniqueName: \"kubernetes.io/projected/5a235b0f-ffbd-4cb2-bc29-19ade07bb186-kube-api-access-bpnb4\") pod \"kserve-module-controller-manager-7566cf5fc8-lrpvq\" (UID: \"5a235b0f-ffbd-4cb2-bc29-19ade07bb186\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.626637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.626560    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:41.805832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.805803    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bd39cb-cdba-435b-921d-94e4da321308" path="/var/lib/kubelet/pods/73bd39cb-cdba-435b-921d-94e4da321308/volumes"
Nov 03 00:10:41.926284 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:10:41.926249    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a235b0f_ffbd_4cb2_bc29_19ade07bb186.slice/crio-37c62c2847ad0cad6b3190f74a650644bb90c529a933ceda0988b8a9daac9dce WatchSource:0}: Error finding container 37c62c2847ad0cad6b3190f74a650644bb90c529a933ceda0988b8a9daac9dce: Status 404 returned error can't find the container with id 37c62c2847ad0cad6b3190f74a650644bb90c529a933ceda0988b8a9daac9dce
Nov 03 00:10:41.926550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:41.926529    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"]
Nov 03 00:10:42.052218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:42.052188    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq" event={"ID":"5a235b0f-ffbd-4cb2-bc29-19ade07bb186","Type":"ContainerStarted","Data":"37c62c2847ad0cad6b3190f74a650644bb90c529a933ceda0988b8a9daac9dce"}
Nov 03 00:10:43.006986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.006948    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 00:10:43.092106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.092071    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"]
Nov 03 00:10:43.092314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.092283    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj" podUID="3a283e6b-0101-4688-b96c-41b549f9f753" containerName="manager" containerID="cri-o://e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570" gracePeriod=10
Nov 03 00:10:43.374815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.374791    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:43.414596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.414559    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl\" (UniqueName: \"kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl\") pod \"3a283e6b-0101-4688-b96c-41b549f9f753\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") "
Nov 03 00:10:43.414730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.414711    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume\") pod \"3a283e6b-0101-4688-b96c-41b549f9f753\" (UID: \"3a283e6b-0101-4688-b96c-41b549f9f753\") "
Nov 03 00:10:43.415049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.415026    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume" pod "3a283e6b-0101-4688-b96c-41b549f9f753" (UID: "3a283e6b-0101-4688-b96c-41b549f9f753"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:10:43.416884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.416856    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl" pod "3a283e6b-0101-4688-b96c-41b549f9f753" (UID: "3a283e6b-0101-4688-b96c-41b549f9f753"). InnerVolumeSpecName "kube-api-access-s8gfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:10:43.515727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.515689    4550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a283e6b-0101-4688-b96c-41b549f9f753-extensions-socket-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:10:43.515727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:43.515727    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gfl\" (UniqueName: \"kubernetes.io/projected/3a283e6b-0101-4688-b96c-41b549f9f753-kube-api-access-s8gfl\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:10:44.062842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.062759    4550 generic.go:356] "Generic (PLEG): container finished" podID="3a283e6b-0101-4688-b96c-41b549f9f753" containerID="e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570" exitCode=0
Nov 03 00:10:44.062842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.062809    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"
Nov 03 00:10:44.063326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.062840    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj" event={"ID":"3a283e6b-0101-4688-b96c-41b549f9f753","Type":"ContainerDied","Data":"e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570"}
Nov 03 00:10:44.063326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.062911    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj" event={"ID":"3a283e6b-0101-4688-b96c-41b549f9f753","Type":"ContainerDied","Data":"18a6a2cecd089268a9fa63cd5a0b2be0bd6234813bb55e4c3fbc8953f9414f48"}
Nov 03 00:10:44.063326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.062941    4550 scope.go:122] "RemoveContainer" containerID="e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570"
Nov 03 00:10:44.071862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.071835    4550 scope.go:122] "RemoveContainer" containerID="e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570"
Nov 03 00:10:44.072170 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:10:44.072146    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570\": container with ID starting with e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570 not found: ID does not exist" containerID="e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570"
Nov 03 00:10:44.072221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.072185    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570"} err="failed to get container status \"e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570\": rpc error: code = NotFound desc = could not find container \"e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570\": container with ID starting with e7325d2f265b7a847e9ee96bc6ea76d05b592db3099b23e69ba15c0483dd7570 not found: ID does not exist"
Nov 03 00:10:44.120683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.120646    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"]
Nov 03 00:10:44.133364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:44.133324    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-m82cj"]
Nov 03 00:10:45.069759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:45.069719    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq" event={"ID":"5a235b0f-ffbd-4cb2-bc29-19ade07bb186","Type":"ContainerStarted","Data":"015d473b1da76dcea6e8f11d34df5bbb51e24bd9b5f2181b7e04ad52a0f095c9"}
Nov 03 00:10:45.070154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:45.069802    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:10:45.124060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:45.124007    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq" podStartSLOduration=1.792162873 podStartE2EDuration="4.123993622s" totalImagesPullingTime="2.331830749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:45.122689047 +0000 UTC m=+327.444225846" watchObservedRunningTime="2026-11-03 00:10:45.123993622 +0000 UTC m=+327.445530421"
Nov 03 00:10:45.805165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:45.805128    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a283e6b-0101-4688-b96c-41b549f9f753" path="/var/lib/kubelet/pods/3a283e6b-0101-4688-b96c-41b549f9f753/volumes"
Nov 03 00:10:47.029008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:47.028978    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:48.762935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:48.762902    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:48.763323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:48.763133    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45" podUID="6975b04f-4fb0-4680-9c99-1d522c39dd04" containerName="limitador" containerID="cri-o://b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3" gracePeriod=30
Nov 03 00:10:49.315225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.315200    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:49.362298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.362214    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file\" (UniqueName: \"kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file\") pod \"6975b04f-4fb0-4680-9c99-1d522c39dd04\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") "
Nov 03 00:10:49.362453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.362407    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9\" (UniqueName: \"kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9\") pod \"6975b04f-4fb0-4680-9c99-1d522c39dd04\" (UID: \"6975b04f-4fb0-4680-9c99-1d522c39dd04\") "
Nov 03 00:10:49.362560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.362538    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file" pod "6975b04f-4fb0-4680-9c99-1d522c39dd04" (UID: "6975b04f-4fb0-4680-9c99-1d522c39dd04"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:10:49.362704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.362691    4550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6975b04f-4fb0-4680-9c99-1d522c39dd04-config-file\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:10:49.364469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.364441    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9" pod "6975b04f-4fb0-4680-9c99-1d522c39dd04" (UID: "6975b04f-4fb0-4680-9c99-1d522c39dd04"). InnerVolumeSpecName "kube-api-access-tn5r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:10:49.463533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:49.463506    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn5r9\" (UniqueName: \"kubernetes.io/projected/6975b04f-4fb0-4680-9c99-1d522c39dd04-kube-api-access-tn5r9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:10:50.090694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.090607    4550 generic.go:356] "Generic (PLEG): container finished" podID="6975b04f-4fb0-4680-9c99-1d522c39dd04" containerID="b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3" exitCode=0
Nov 03 00:10:50.090694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.090643    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45" event={"ID":"6975b04f-4fb0-4680-9c99-1d522c39dd04","Type":"ContainerDied","Data":"b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3"}
Nov 03 00:10:50.090694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.090653    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45"
Nov 03 00:10:50.090694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.090686    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xnl45" event={"ID":"6975b04f-4fb0-4680-9c99-1d522c39dd04","Type":"ContainerDied","Data":"e574d387f5690a396761ee12f60d1bb64aa8fce53349aab4328b8846a4d4eff2"}
Nov 03 00:10:50.091207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.090707    4550 scope.go:122] "RemoveContainer" containerID="b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3"
Nov 03 00:10:50.098351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.098334    4550 scope.go:122] "RemoveContainer" containerID="b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3"
Nov 03 00:10:50.098599 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:10:50.098580    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3\": container with ID starting with b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3 not found: ID does not exist" containerID="b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3"
Nov 03 00:10:50.098649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.098606    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3"} err="failed to get container status \"b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3\": rpc error: code = NotFound desc = could not find container \"b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3\": container with ID starting with b2782ba2f2649af843d1c80cbc53d5215e2e5717c84baeab98b57d8c1a9c58e3 not found: ID does not exist"
Nov 03 00:10:50.141479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.141447    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:50.154353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:50.154330    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xnl45"]
Nov 03 00:10:51.808013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:10:51.807978    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6975b04f-4fb0-4680-9c99-1d522c39dd04" path="/var/lib/kubelet/pods/6975b04f-4fb0-4680-9c99-1d522c39dd04/volumes"
Nov 03 00:10:54.721682 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:10:54.721646    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:11:04.466739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:04.466703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:04.574741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:04.574702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:05.104666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.104637    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"]
Nov 03 00:11:05.104981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.104968    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3a283e6b-0101-4688-b96c-41b549f9f753" containerName="manager"
Nov 03 00:11:05.105031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.104983    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a283e6b-0101-4688-b96c-41b549f9f753" containerName="manager"
Nov 03 00:11:05.105031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.104996    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6975b04f-4fb0-4680-9c99-1d522c39dd04" containerName="limitador"
Nov 03 00:11:05.105031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.105001    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6975b04f-4fb0-4680-9c99-1d522c39dd04" containerName="limitador"
Nov 03 00:11:05.105119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.105064    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a283e6b-0101-4688-b96c-41b549f9f753" containerName="manager"
Nov 03 00:11:05.105119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.105073    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6975b04f-4fb0-4680-9c99-1d522c39dd04" containerName="limitador"
Nov 03 00:11:05.159654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.159623    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"]
Nov 03 00:11:05.159811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.159736    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.175449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.175411    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:11:05.175588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.175574    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:11:05.184903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.184877    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-92wjz\""
Nov 03 00:11:05.184903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.184883    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 00:11:05.186846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.186824    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:05.302271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.302228    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ce8b481-b56a-48a3-a84c-9bf39b36d2ff-data\") pod \"postgres-7cdbf6dc44-mr4ms\" (UID: \"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.302474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.302309    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92rj7\" (UniqueName: \"kubernetes.io/projected/9ce8b481-b56a-48a3-a84c-9bf39b36d2ff-kube-api-access-92rj7\") pod \"postgres-7cdbf6dc44-mr4ms\" (UID: \"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.402746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.402718    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ce8b481-b56a-48a3-a84c-9bf39b36d2ff-data\") pod \"postgres-7cdbf6dc44-mr4ms\" (UID: \"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.402908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.402775    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92rj7\" (UniqueName: \"kubernetes.io/projected/9ce8b481-b56a-48a3-a84c-9bf39b36d2ff-kube-api-access-92rj7\") pod \"postgres-7cdbf6dc44-mr4ms\" (UID: \"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.403122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.403100    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9ce8b481-b56a-48a3-a84c-9bf39b36d2ff-data\") pod \"postgres-7cdbf6dc44-mr4ms\" (UID: \"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.422389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.422358    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92rj7\" (UniqueName: \"kubernetes.io/projected/9ce8b481-b56a-48a3-a84c-9bf39b36d2ff-kube-api-access-92rj7\") pod \"postgres-7cdbf6dc44-mr4ms\" (UID: \"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.483341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.483316    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:05.794242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:05.794211    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"]
Nov 03 00:11:05.794747 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:11:05.794719    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce8b481_b56a_48a3_a84c_9bf39b36d2ff.slice/crio-ba27201ee495469e68d7abf1a38516c50f9b49a230415fea9873fc75cf5d13d6 WatchSource:0}: Error finding container ba27201ee495469e68d7abf1a38516c50f9b49a230415fea9873fc75cf5d13d6: Status 404 returned error can't find the container with id ba27201ee495469e68d7abf1a38516c50f9b49a230415fea9873fc75cf5d13d6
Nov 03 00:11:06.074920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:06.074846    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-lrpvq"
Nov 03 00:11:06.157727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:06.157693    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms" event={"ID":"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff","Type":"ContainerStarted","Data":"ba27201ee495469e68d7abf1a38516c50f9b49a230415fea9873fc75cf5d13d6"}
Nov 03 00:11:07.939249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:07.939168    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:11.180715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:11.180676    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms" event={"ID":"9ce8b481-b56a-48a3-a84c-9bf39b36d2ff","Type":"ContainerStarted","Data":"29223bfba809177254840954d086109108654eb6c3e2cc98926b50971f8ef6f3"}
Nov 03 00:11:11.181167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:11.180741    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:11.220313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:11.220262    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms" podStartSLOduration=1.229664154 podStartE2EDuration="6.220248621s" totalImagesPullingTime="4.990584467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:11.219974867 +0000 UTC m=+353.541511650" watchObservedRunningTime="2026-11-03 00:11:11.220248621 +0000 UTC m=+353.541785421"
Nov 03 00:11:17.215558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:17.215524    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-mr4ms"
Nov 03 00:11:23.778039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:23.777966    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:23.879014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:23.878969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:23.978795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:23.978759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:24.078861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:24.078771    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:24.721808 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:11:24.721767    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:11:27.374529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:27.374487    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:27.484526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:27.484495    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:29.096895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:29.096857    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:29.197455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:29.197392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:36.982259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:36.982221    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.084080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.083974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.183679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.183634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.224033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.223994    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2hwfp"]
Nov 03 00:11:37.231010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.230983    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.245397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.245366    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2hwfp"]
Nov 03 00:11:37.258830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.258792    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-czh75\""
Nov 03 00:11:37.258830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.258802    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 00:11:37.273056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.273021    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"]
Nov 03 00:11:37.273349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.273323    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bmn\" (UniqueName: \"kubernetes.io/projected/8588e986-19d2-4ce9-9c19-fc2ae2859e26-kube-api-access-p5bmn\") pod \"kserve-controller-manager-776cc66d47-2hwfp\" (UID: \"8588e986-19d2-4ce9-9c19-fc2ae2859e26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.273501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.273471    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8588e986-19d2-4ce9-9c19-fc2ae2859e26-cert\") pod \"kserve-controller-manager-776cc66d47-2hwfp\" (UID: \"8588e986-19d2-4ce9-9c19-fc2ae2859e26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.276562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.276539    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.300499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.297601    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"]
Nov 03 00:11:37.304823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.304787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.310257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.310223    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hfkg4"]
Nov 03 00:11:37.313707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.313683    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.334087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.334054    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hfkg4"]
Nov 03 00:11:37.346727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.346693    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 00:11:37.346909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.346752    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-g8gc2\""
Nov 03 00:11:37.346909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.346860    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 00:11:37.346989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.346909    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j74sd\""
Nov 03 00:11:37.374311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.374259    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8588e986-19d2-4ce9-9c19-fc2ae2859e26-cert\") pod \"kserve-controller-manager-776cc66d47-2hwfp\" (UID: \"8588e986-19d2-4ce9-9c19-fc2ae2859e26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.374545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.374356    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtfp7\" (UniqueName: \"kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7\") pod \"llmisvc-controller-manager-869d5dcc5d-4zp9q\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.374545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.374389    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bmn\" (UniqueName: \"kubernetes.io/projected/8588e986-19d2-4ce9-9c19-fc2ae2859e26-kube-api-access-p5bmn\") pod \"kserve-controller-manager-776cc66d47-2hwfp\" (UID: \"8588e986-19d2-4ce9-9c19-fc2ae2859e26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.374545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.374413    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c24118b-7393-4ad0-96ae-11d71d2dad5b-tls-certs\") pod \"model-serving-api-75bd89fdfc-hfkg4\" (UID: \"4c24118b-7393-4ad0-96ae-11d71d2dad5b\") " pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.374545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.374498    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5jx\" (UniqueName: \"kubernetes.io/projected/4c24118b-7393-4ad0-96ae-11d71d2dad5b-kube-api-access-9g5jx\") pod \"model-serving-api-75bd89fdfc-hfkg4\" (UID: \"4c24118b-7393-4ad0-96ae-11d71d2dad5b\") " pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.374721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.374576    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4zp9q\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.377149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.377116    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8588e986-19d2-4ce9-9c19-fc2ae2859e26-cert\") pod \"kserve-controller-manager-776cc66d47-2hwfp\" (UID: \"8588e986-19d2-4ce9-9c19-fc2ae2859e26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.394606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.394568    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bmn\" (UniqueName: \"kubernetes.io/projected/8588e986-19d2-4ce9-9c19-fc2ae2859e26-kube-api-access-p5bmn\") pod \"kserve-controller-manager-776cc66d47-2hwfp\" (UID: \"8588e986-19d2-4ce9-9c19-fc2ae2859e26\") " pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.396941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.396908    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.475529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.475480    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4zp9q\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.475711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.475613    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtfp7\" (UniqueName: \"kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7\") pod \"llmisvc-controller-manager-869d5dcc5d-4zp9q\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.475711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.475645    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c24118b-7393-4ad0-96ae-11d71d2dad5b-tls-certs\") pod \"model-serving-api-75bd89fdfc-hfkg4\" (UID: \"4c24118b-7393-4ad0-96ae-11d71d2dad5b\") " pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.475711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.475669    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5jx\" (UniqueName: \"kubernetes.io/projected/4c24118b-7393-4ad0-96ae-11d71d2dad5b-kube-api-access-9g5jx\") pod \"model-serving-api-75bd89fdfc-hfkg4\" (UID: \"4c24118b-7393-4ad0-96ae-11d71d2dad5b\") " pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.478186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.478154    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c24118b-7393-4ad0-96ae-11d71d2dad5b-tls-certs\") pod \"model-serving-api-75bd89fdfc-hfkg4\" (UID: \"4c24118b-7393-4ad0-96ae-11d71d2dad5b\") " pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.478186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.478179    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-4zp9q\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.496005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.495919    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtfp7\" (UniqueName: \"kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7\") pod \"llmisvc-controller-manager-869d5dcc5d-4zp9q\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.496005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.495993    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5jx\" (UniqueName: \"kubernetes.io/projected/4c24118b-7393-4ad0-96ae-11d71d2dad5b-kube-api-access-9g5jx\") pod \"model-serving-api-75bd89fdfc-hfkg4\" (UID: \"4c24118b-7393-4ad0-96ae-11d71d2dad5b\") " pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.497908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.497870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.544958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.544926    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:37.589893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.589849    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:37.628142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.628093    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:37.717918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.717879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:37.843354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.843321    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-2hwfp"]
Nov 03 00:11:37.843904 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:11:37.843871    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8588e986_19d2_4ce9_9c19_fc2ae2859e26.slice/crio-6fd7e9132a3541494ef9c79568b3f60c6adc24051f5167577ab1efea8295bf6d WatchSource:0}: Error finding container 6fd7e9132a3541494ef9c79568b3f60c6adc24051f5167577ab1efea8295bf6d: Status 404 returned error can't find the container with id 6fd7e9132a3541494ef9c79568b3f60c6adc24051f5167577ab1efea8295bf6d
Nov 03 00:11:37.901505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.901479    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"]
Nov 03 00:11:37.901945 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:11:37.901916    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6aa639fe_2f07_4330_bd2e_5ba36bc44619.slice/crio-8a601b2c06abda17cd6f3b2ccfe9843bd894173038a3ec0ff78eb0a359e14229 WatchSource:0}: Error finding container 8a601b2c06abda17cd6f3b2ccfe9843bd894173038a3ec0ff78eb0a359e14229: Status 404 returned error can't find the container with id 8a601b2c06abda17cd6f3b2ccfe9843bd894173038a3ec0ff78eb0a359e14229
Nov 03 00:11:37.931866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:37.931838    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-hfkg4"]
Nov 03 00:11:37.932326 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:11:37.932301    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c24118b_7393_4ad0_96ae_11d71d2dad5b.slice/crio-1869487f573fdbce7ac4d958b48400315b12c868214cd8bff1e4805c6654a3a9 WatchSource:0}: Error finding container 1869487f573fdbce7ac4d958b48400315b12c868214cd8bff1e4805c6654a3a9: Status 404 returned error can't find the container with id 1869487f573fdbce7ac4d958b48400315b12c868214cd8bff1e4805c6654a3a9
Nov 03 00:11:38.291717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:38.291673    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q" event={"ID":"6aa639fe-2f07-4330-bd2e-5ba36bc44619","Type":"ContainerStarted","Data":"8a601b2c06abda17cd6f3b2ccfe9843bd894173038a3ec0ff78eb0a359e14229"}
Nov 03 00:11:38.292904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:38.292874    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4" event={"ID":"4c24118b-7393-4ad0-96ae-11d71d2dad5b","Type":"ContainerStarted","Data":"1869487f573fdbce7ac4d958b48400315b12c868214cd8bff1e4805c6654a3a9"}
Nov 03 00:11:38.293927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:38.293905    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp" event={"ID":"8588e986-19d2-4ce9-9c19-fc2ae2859e26","Type":"ContainerStarted","Data":"6fd7e9132a3541494ef9c79568b3f60c6adc24051f5167577ab1efea8295bf6d"}
Nov 03 00:11:39.552826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:39.552786    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:39.658450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:39.658393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:39.759149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:39.759031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:39.858629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:39.858470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:39.959264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:39.959232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:40.307661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:40.307624    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4" event={"ID":"4c24118b-7393-4ad0-96ae-11d71d2dad5b","Type":"ContainerStarted","Data":"fd284e5a36cda7d646a9cff30b9873ad0a6fddf736a2af7b5c6d47daaab319bb"}
Nov 03 00:11:40.307803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:40.307699    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:40.346628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:40.346578    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4" podStartSLOduration=1.324650037 podStartE2EDuration="3.346563581s" totalImagesPullingTime="2.021913544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:40.345622978 +0000 UTC m=+382.667159799" watchObservedRunningTime="2026-11-03 00:11:40.346563581 +0000 UTC m=+382.668100380"
Nov 03 00:11:41.313516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:41.313468    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp" event={"ID":"8588e986-19d2-4ce9-9c19-fc2ae2859e26","Type":"ContainerStarted","Data":"1b199f7d2ec8a52607ce96d85e568f89bbcc3b30f8d7ac3236f727dbb2e2eab0"}
Nov 03 00:11:41.313998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:41.313676    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:11:41.315145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:41.315114    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q" event={"ID":"6aa639fe-2f07-4330-bd2e-5ba36bc44619","Type":"ContainerStarted","Data":"1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8"}
Nov 03 00:11:41.315286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:41.315216    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:11:41.390641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:41.390575    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp" podStartSLOduration=1.988132905 podStartE2EDuration="4.390558366s" totalImagesPullingTime="2.402425461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:41.351876539 +0000 UTC m=+383.673413339" watchObservedRunningTime="2026-11-03 00:11:41.390558366 +0000 UTC m=+383.712095165"
Nov 03 00:11:41.391082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:41.391057    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q" podStartSLOduration=2.0073071 podStartE2EDuration="4.391049175s" totalImagesPullingTime="2.383742075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:41.389384675 +0000 UTC m=+383.710921474" watchObservedRunningTime="2026-11-03 00:11:41.391049175 +0000 UTC m=+383.712586008"
Nov 03 00:11:45.689894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:45.689860    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:45.883247 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:45.883209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:46.293841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.293796    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"]
Nov 03 00:11:46.315149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.315106    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.317812 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.317780    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"]
Nov 03 00:11:46.333504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.333467    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 00:11:46.346792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.346760    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 00:11:46.346792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.346770    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-nqtv9\""
Nov 03 00:11:46.346995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.346822    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 00:11:46.357904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.357870    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-metrics-certs\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.357904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.357905    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-manager-config\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.358148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.357993    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-cert\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.358148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.358064    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnwg\" (UniqueName: \"kubernetes.io/projected/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-kube-api-access-2bnwg\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.459319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.459278    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-metrics-certs\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.459319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.459321    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-manager-config\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.459675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.459363    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-cert\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.459675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.459462    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnwg\" (UniqueName: \"kubernetes.io/projected/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-kube-api-access-2bnwg\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.460087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.460057    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-manager-config\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.461989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.461963    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-metrics-certs\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.462102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.461989    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-cert\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.481487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.481451    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnwg\" (UniqueName: \"kubernetes.io/projected/524c85b9-2b14-4c2d-ad51-3e2db2c4899b-kube-api-access-2bnwg\") pod \"jobset-controller-manager-594589c6fc-nmgpm\" (UID: \"524c85b9-2b14-4c2d-ad51-3e2db2c4899b\") " pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.631259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.631165    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:46.927663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:46.927632    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"]
Nov 03 00:11:46.928057 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:11:46.927897    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524c85b9_2b14_4c2d_ad51_3e2db2c4899b.slice/crio-66bdd501abe4baa8f68bf5dcfc8b59f57d7ae8db95464c3c9de258cce2eae8b2 WatchSource:0}: Error finding container 66bdd501abe4baa8f68bf5dcfc8b59f57d7ae8db95464c3c9de258cce2eae8b2: Status 404 returned error can't find the container with id 66bdd501abe4baa8f68bf5dcfc8b59f57d7ae8db95464c3c9de258cce2eae8b2
Nov 03 00:11:47.341503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:47.341391    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm" event={"ID":"524c85b9-2b14-4c2d-ad51-3e2db2c4899b","Type":"ContainerStarted","Data":"66bdd501abe4baa8f68bf5dcfc8b59f57d7ae8db95464c3c9de258cce2eae8b2"}
Nov 03 00:11:51.320036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:51.320000    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-hfkg4"
Nov 03 00:11:51.361942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:51.361904    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm" event={"ID":"524c85b9-2b14-4c2d-ad51-3e2db2c4899b","Type":"ContainerStarted","Data":"ea50b95fbc5ac41a54c181436ff3e9895a741d2a84fba77aa680f5d0030824c1"}
Nov 03 00:11:51.362144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:51.362025    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:11:51.402286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:51.402220    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm" podStartSLOduration=1.465208715 podStartE2EDuration="5.402202552s" totalImagesPullingTime="3.936993837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:51.401291887 +0000 UTC m=+393.722828689" watchObservedRunningTime="2026-11-03 00:11:51.402202552 +0000 UTC m=+393.723739352"
Nov 03 00:11:54.724030 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:11:54.723992    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:11:58.740221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:58.740185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:11:58.841117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:11:58.841076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:00.229829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:00.229750    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:00.329486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:00.329448    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:00.430585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:00.430540    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:02.371968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:02.371938    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-594589c6fc-nmgpm"
Nov 03 00:12:05.463236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:05.463204    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:08.077762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:08.077726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:11.553720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:11.553686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:11.757686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:11.757648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:12:12.321397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:12.321370    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:12:12.325023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:12.325004    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-2hwfp"
Nov 03 00:12:24.721846 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:12:24.721801    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:12:35.431649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.431612    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p876r"]
Nov 03 00:12:35.435094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.435073    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.451288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.451247    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1de0029a-6ab4-47db-9562-805aab89a6ea-cert\") pod \"llmisvc-controller-manager-6466c554d8-p876r\" (UID: \"1de0029a-6ab4-47db-9562-805aab89a6ea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.451798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.451760    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpd2m\" (UniqueName: \"kubernetes.io/projected/1de0029a-6ab4-47db-9562-805aab89a6ea-kube-api-access-dpd2m\") pod \"llmisvc-controller-manager-6466c554d8-p876r\" (UID: \"1de0029a-6ab4-47db-9562-805aab89a6ea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.453232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.453205    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p876r"]
Nov 03 00:12:35.552295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.552255    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpd2m\" (UniqueName: \"kubernetes.io/projected/1de0029a-6ab4-47db-9562-805aab89a6ea-kube-api-access-dpd2m\") pod \"llmisvc-controller-manager-6466c554d8-p876r\" (UID: \"1de0029a-6ab4-47db-9562-805aab89a6ea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.552537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.552309    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1de0029a-6ab4-47db-9562-805aab89a6ea-cert\") pod \"llmisvc-controller-manager-6466c554d8-p876r\" (UID: \"1de0029a-6ab4-47db-9562-805aab89a6ea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.555048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.555018    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1de0029a-6ab4-47db-9562-805aab89a6ea-cert\") pod \"llmisvc-controller-manager-6466c554d8-p876r\" (UID: \"1de0029a-6ab4-47db-9562-805aab89a6ea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.572056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.572022    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpd2m\" (UniqueName: \"kubernetes.io/projected/1de0029a-6ab4-47db-9562-805aab89a6ea-kube-api-access-dpd2m\") pod \"llmisvc-controller-manager-6466c554d8-p876r\" (UID: \"1de0029a-6ab4-47db-9562-805aab89a6ea\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:35.747830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:35.747719    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:36.041079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:36.041048    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-p876r"]
Nov 03 00:12:36.041400 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:12:36.041376    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de0029a_6ab4_47db_9562_805aab89a6ea.slice/crio-de022b782046f3262b7eb088efb4f92d3d156ebe35863fe8b512f0ada4f432e3 WatchSource:0}: Error finding container de022b782046f3262b7eb088efb4f92d3d156ebe35863fe8b512f0ada4f432e3: Status 404 returned error can't find the container with id de022b782046f3262b7eb088efb4f92d3d156ebe35863fe8b512f0ada4f432e3
Nov 03 00:12:36.548282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:36.548246    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r" event={"ID":"1de0029a-6ab4-47db-9562-805aab89a6ea","Type":"ContainerStarted","Data":"08120e2badabb616464e85fe679b1b9465b216378a24e7afdce84391b2a8b027"}
Nov 03 00:12:36.548741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:36.548303    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r" event={"ID":"1de0029a-6ab4-47db-9562-805aab89a6ea","Type":"ContainerStarted","Data":"de022b782046f3262b7eb088efb4f92d3d156ebe35863fe8b512f0ada4f432e3"}
Nov 03 00:12:36.548741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:36.548349    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:12:36.587852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:12:36.587746    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r" podStartSLOduration=1.310444695 podStartE2EDuration="1.587730475s" totalImagesPullingTime="277.28578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:36.587229299 +0000 UTC m=+438.908766099" watchObservedRunningTime="2026-11-03 00:12:36.587730475 +0000 UTC m=+438.909267274"
Nov 03 00:12:54.723783 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:12:54.723743    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:13:04.323560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:04.323517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:04.410414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:04.410373    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:07.555652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.555621    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-p876r"
Nov 03 00:13:07.623578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.623542    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"]
Nov 03 00:13:07.623781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.623755    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q" podUID="6aa639fe-2f07-4330-bd2e-5ba36bc44619" containerName="manager" containerID="cri-o://1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8" gracePeriod=30
Nov 03 00:13:07.878091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.878065    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:13:07.934852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.934809    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert\" (UniqueName: \"kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert\") pod \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") "
Nov 03 00:13:07.935054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.934885    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7\" (UniqueName: \"kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7\") pod \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\" (UID: \"6aa639fe-2f07-4330-bd2e-5ba36bc44619\") "
Nov 03 00:13:07.937127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.937092    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert" pod "6aa639fe-2f07-4330-bd2e-5ba36bc44619" (UID: "6aa639fe-2f07-4330-bd2e-5ba36bc44619"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:13:07.937274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:07.937185    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7" pod "6aa639fe-2f07-4330-bd2e-5ba36bc44619" (UID: "6aa639fe-2f07-4330-bd2e-5ba36bc44619"). InnerVolumeSpecName "kube-api-access-gtfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:13:08.035659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.035622    4550 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6aa639fe-2f07-4330-bd2e-5ba36bc44619-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:13:08.035659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.035653    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtfp7\" (UniqueName: \"kubernetes.io/projected/6aa639fe-2f07-4330-bd2e-5ba36bc44619-kube-api-access-gtfp7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:13:08.186853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.186813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:08.685149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.685113    4550 generic.go:356] "Generic (PLEG): container finished" podID="6aa639fe-2f07-4330-bd2e-5ba36bc44619" containerID="1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8" exitCode=0
Nov 03 00:13:08.685554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.685164    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"
Nov 03 00:13:08.685554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.685157    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q" event={"ID":"6aa639fe-2f07-4330-bd2e-5ba36bc44619","Type":"ContainerDied","Data":"1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8"}
Nov 03 00:13:08.685554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.685288    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q" event={"ID":"6aa639fe-2f07-4330-bd2e-5ba36bc44619","Type":"ContainerDied","Data":"8a601b2c06abda17cd6f3b2ccfe9843bd894173038a3ec0ff78eb0a359e14229"}
Nov 03 00:13:08.685554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.685311    4550 scope.go:122] "RemoveContainer" containerID="1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8"
Nov 03 00:13:08.693733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.693716    4550 scope.go:122] "RemoveContainer" containerID="1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8"
Nov 03 00:13:08.694024 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:13:08.694004    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8\": container with ID starting with 1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8 not found: ID does not exist" containerID="1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8"
Nov 03 00:13:08.694099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.694031    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8"} err="failed to get container status \"1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8\": rpc error: code = NotFound desc = could not find container \"1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8\": container with ID starting with 1dc0d0512778543762addf496db6d81e47dd9ed6c6ef0e2e082d0bfe36f060c8 not found: ID does not exist"
Nov 03 00:13:08.736443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.736403    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"]
Nov 03 00:13:08.749714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.749692    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-4zp9q"]
Nov 03 00:13:08.811771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.811736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:08.939705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:08.939616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:09.806135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:09.806101    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa639fe-2f07-4330-bd2e-5ba36bc44619" path="/var/lib/kubelet/pods/6aa639fe-2f07-4330-bd2e-5ba36bc44619/volumes"
Nov 03 00:13:13.106162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:13.106127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:13.203000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:13.202963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:24.721207 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:13:24.721162    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:13:32.428610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:32.428528    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:32.523598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:32.523560    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:37.153672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:37.153630    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:37.659291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:37.659260    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:38.882190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:38.882149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:38.992686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:38.992636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:43.238022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:43.237988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:46.828224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:46.828191    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:48.219714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:48.219677    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:49.369965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:49.369923    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:51.340714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:51.340683    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:52.877551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:52.877518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:54.722540 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:13:54.722499    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:13:55.225644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:55.225607    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:13:58.000850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:13:58.000816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:08.290669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:08.290630    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:12.214314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.214282    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"]
Nov 03 00:14:12.214702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.214660    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6aa639fe-2f07-4330-bd2e-5ba36bc44619" containerName="manager"
Nov 03 00:14:12.214702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.214672    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa639fe-2f07-4330-bd2e-5ba36bc44619" containerName="manager"
Nov 03 00:14:12.214776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.214730    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa639fe-2f07-4330-bd2e-5ba36bc44619" containerName="manager"
Nov 03 00:14:12.217450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.217422    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.239150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.239117    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 00:14:12.249042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.249012    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 00:14:12.249223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.249186    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 00:14:12.250522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.250501    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"]
Nov 03 00:14:12.319817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.319777    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v9b8n\""
Nov 03 00:14:12.389860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.389819    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jq7\" (UniqueName: \"kubernetes.io/projected/caac059c-fc5f-4b53-99d5-c98bccf3480d-kube-api-access-k8jq7\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.389984    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390045    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac059c-fc5f-4b53-99d5-c98bccf3480d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390157    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390187    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390268    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390308    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390338    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.390474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.390388    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.491644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.491554    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.491644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.491595    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jq7\" (UniqueName: \"kubernetes.io/projected/caac059c-fc5f-4b53-99d5-c98bccf3480d-kube-api-access-k8jq7\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.491877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.491658    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.491877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.491685    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac059c-fc5f-4b53-99d5-c98bccf3480d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.491877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.491708    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.491877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.491842    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492046    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492088    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492144    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492171    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492628    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492460    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.492790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.492541    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/caac059c-fc5f-4b53-99d5-c98bccf3480d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.494706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.494676    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.494706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.494688    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.512175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.512145    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/caac059c-fc5f-4b53-99d5-c98bccf3480d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.512290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.512241    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jq7\" (UniqueName: \"kubernetes.io/projected/caac059c-fc5f-4b53-99d5-c98bccf3480d-kube-api-access-k8jq7\") pod \"maas-default-gateway-openshift-default-6cc458d887-rcjbb\" (UID: \"caac059c-fc5f-4b53-99d5-c98bccf3480d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.533202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.533172    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:12.845325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.845296    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"]
Nov 03 00:14:12.845325 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:14:12.845304    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaac059c_fc5f_4b53_99d5_c98bccf3480d.slice/crio-163fb3b08e34e9370dcec9a0f20702c3a00366db872c1313403c21e2b8ae7531 WatchSource:0}: Error finding container 163fb3b08e34e9370dcec9a0f20702c3a00366db872c1313403c21e2b8ae7531: Status 404 returned error can't find the container with id 163fb3b08e34e9370dcec9a0f20702c3a00366db872c1313403c21e2b8ae7531
Nov 03 00:14:12.966512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:12.966473    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" event={"ID":"caac059c-fc5f-4b53-99d5-c98bccf3480d","Type":"ContainerStarted","Data":"163fb3b08e34e9370dcec9a0f20702c3a00366db872c1313403c21e2b8ae7531"}
Nov 03 00:14:15.853842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:15.853811    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:15.854146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:15.853877    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:15.854146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:15.853907    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:15.982878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:15.982832    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" event={"ID":"caac059c-fc5f-4b53-99d5-c98bccf3480d","Type":"ContainerStarted","Data":"f4eb9c288d9b6bcce83b52b44392783f47184857b2037710894c3736797983c7"}
Nov 03 00:14:16.053231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:16.053170    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podStartSLOduration=1.04769352 podStartE2EDuration="4.05315559s" totalImagesPullingTime="3.00546207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:16.038905597 +0000 UTC m=+538.360442398" watchObservedRunningTime="2026-11-03 00:14:16.05315559 +0000 UTC m=+538.374692389"
Nov 03 00:14:16.533367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:16.533328    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:16.534969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:16.534939    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:16.535101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:16.534993    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:17.534038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:17.534000    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:17.534469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:17.534059    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:18.534092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:18.534046    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:18.534529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:18.534116    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:19.534190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:19.534141    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:19.534619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:19.534214    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:20.533760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:20.533721    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:20.533941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:20.533776    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:21.533777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:21.533737    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:21.534213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:21.533795    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:22.073889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:22.073849    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:22.192961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:22.192927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:22.533662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:22.533627    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Nov 03 00:14:22.533847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:22.533684    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Nov 03 00:14:23.596030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:23.595973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:23.700169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:23.700115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:24.535010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:24.534973    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:14:24.535208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:24.535026    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:14:24.725631 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:14:24.725592    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:14:25.576657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:25.576612    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:14:25.576854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:25.576686    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:14:26.521058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:26.521025    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:26.521565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:26.521370    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:26.522167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:26.522145    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb"
Nov 03 00:14:27.261362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.261327    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"]
Nov 03 00:14:27.265236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.265204    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.295740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.295706    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"]
Nov 03 00:14:27.346273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346235    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346318    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346344    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346446    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346541    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346603    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346635    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbm9\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346706    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.346817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.346749    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448042    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448082    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448102    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbm9\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448133    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448156    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448240    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448264    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448287    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448319    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448645    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448722    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448754    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448845    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.448962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.448938    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.450487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.450460    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.450835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.450819    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.467790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.467760    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.468261 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.468237    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbm9\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9\") pod \"maas-default-gateway-openshift-default-6cc458d887-lnm9s\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.578854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.578752    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:27.885108 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:14:27.885062    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f0e546_9de9_4332_830d_c6e742938d4e.slice/crio-8d72558cbf3a24dbe75926c92b575f492bbbba110a2dde15371afaaf8e395948 WatchSource:0}: Error finding container 8d72558cbf3a24dbe75926c92b575f492bbbba110a2dde15371afaaf8e395948: Status 404 returned error can't find the container with id 8d72558cbf3a24dbe75926c92b575f492bbbba110a2dde15371afaaf8e395948
Nov 03 00:14:27.885795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.885766    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"]
Nov 03 00:14:27.887488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.887455    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:27.887577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.887521    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:27.887577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:27.887551    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:14:28.040885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:28.040847    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" event={"ID":"c8f0e546-9de9-4332-830d-c6e742938d4e","Type":"ContainerStarted","Data":"6a2aa14226a8c3c4a238aa297f10a999a432eec88ea9a9727547d413b12456d6"}
Nov 03 00:14:28.041073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:28.040914    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" event={"ID":"c8f0e546-9de9-4332-830d-c6e742938d4e","Type":"ContainerStarted","Data":"8d72558cbf3a24dbe75926c92b575f492bbbba110a2dde15371afaaf8e395948"}
Nov 03 00:14:28.086212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:28.086140    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podStartSLOduration=1.086119005 podStartE2EDuration="1.086119005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:28.084215931 +0000 UTC m=+550.405752730" watchObservedRunningTime="2026-11-03 00:14:28.086119005 +0000 UTC m=+550.407655807"
Nov 03 00:14:28.583948 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:28.581826    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:28.587680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:28.587582    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:28.587680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:28.587667    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:29.580457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:29.579864    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:29.580457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:29.579929    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:30.580895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:30.580857    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:30.581318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:30.580914    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:31.580878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:31.580837    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:31.581068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:31.580896    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:32.580942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:32.580896    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:32.581132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:32.580974    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:33.580860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:33.580822    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:33.581067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:33.580880    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:34.580152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:34.580111    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Nov 03 00:14:34.580677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:34.580175    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Nov 03 00:14:36.195461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:36.195407    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:36.195986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:36.195848    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:36.196706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:36.196684    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:14:42.610982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:42.610944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:54.722807 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:14:54.722769    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:14:58.601053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:58.601013    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:58.702008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:58.701971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:58.820236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:58.820196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:58.923637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:58.923598    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.183547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:59.183460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.332891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:59.332854    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.441528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:59.440995    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.547714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:59.547665    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.658467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:59.658183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:14:59.783355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:14:59.783268    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:00.062307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:00.062207    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:00.439926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:00.439883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:00.926927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:00.926886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:01.042856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:01.042820    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:01.891116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:01.891080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:02.016153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:02.016117    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:02.702692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:02.702656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:02.865293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:02.865253    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:03.509979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:03.509943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:03.673284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:03.673248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:03.777044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:03.776956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:03.915832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:03.915798    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:04.022852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:04.022820    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:04.397776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:04.397737    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:04.503324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:04.503286    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:04.632451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:04.632403    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:05.367151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:05.367117    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:05.475021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:05.474985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:05.786460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:05.786404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:05.895452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:05.891668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:06.414340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:06.414304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:06.532766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:06.532718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:07.272277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:07.272225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:07.375979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:07.375942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:07.567767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:07.567670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:07.667341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:07.667304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:08.075697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:08.075661    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:08.244889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:08.244848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:08.413653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:08.413618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:08.603716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:08.603679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:08.703767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:08.703675    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:09.507700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:09.507662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:09.606773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:09.606738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:09.708865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:09.708827    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:11.117001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:11.116965    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:11.237183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:11.237149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:11.647519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:11.647479    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:12.773724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:12.773678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:12.868652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:12.868606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:13.435385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:13.435342    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:13.530693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:13.530645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:13.631538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:13.631500    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:15.384236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:15.384195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:16.962768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:16.962727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:17.979248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:17.979208    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"]
Nov 03 00:15:17.982732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:17.982713    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.013803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.013778    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:15:18.013987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.013951    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:15:18.016366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.016337    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"]
Nov 03 00:15:18.113763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.113727    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"]
Nov 03 00:15:18.117167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.117149    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.125798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.125774    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnstn\" (UniqueName: \"kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.125914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.125843    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.125914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.125886    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.134377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.134348    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"]
Nov 03 00:15:18.226771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.226736    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.226955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.226789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfkq\" (UniqueName: \"kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.226955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.226809    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.227039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.226967    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.227039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.227005    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.227105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.227052    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnstn\" (UniqueName: \"kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.227593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.227569    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.229320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.229269    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.246758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.246733    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnstn\" (UniqueName: \"kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn\") pod \"maas-api-revoke-keys-e2e-ait-c600f063-wlqr4\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.295220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.295186    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:18.327767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.327738    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfkq\" (UniqueName: \"kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.327892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.327774    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.327892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.327801    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.328658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.328630    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.330120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.330101    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.351597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.351006    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfkq\" (UniqueName: \"kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq\") pod \"maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.429842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.429810    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:18.593142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.593103    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"]
Nov 03 00:15:18.593588 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:15:18.593550    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3568d5_6a63_43db_8cf6_998f232cd0d6.slice/crio-d1e97736cb393f44bc6b7538cdff396e704c756078dd6cb8d708063e506d9f59 WatchSource:0}: Error finding container d1e97736cb393f44bc6b7538cdff396e704c756078dd6cb8d708063e506d9f59: Status 404 returned error can't find the container with id d1e97736cb393f44bc6b7538cdff396e704c756078dd6cb8d708063e506d9f59
Nov 03 00:15:18.732380 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:15:18.732340    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d99ed2_1f12_4bb9_a672_908fcf0a9ab3.slice/crio-ae2498c0b2c98073b4265a2e83f1b8a731a28b5c05c7c331fc5bd21d28b38ecd WatchSource:0}: Error finding container ae2498c0b2c98073b4265a2e83f1b8a731a28b5c05c7c331fc5bd21d28b38ecd: Status 404 returned error can't find the container with id ae2498c0b2c98073b4265a2e83f1b8a731a28b5c05c7c331fc5bd21d28b38ecd
Nov 03 00:15:18.732784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:18.732759    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"]
Nov 03 00:15:19.134097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:19.134064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:19.279145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:19.279105    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c" event={"ID":"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3","Type":"ContainerStarted","Data":"ae2498c0b2c98073b4265a2e83f1b8a731a28b5c05c7c331fc5bd21d28b38ecd"}
Nov 03 00:15:19.280209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:19.280186    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4" event={"ID":"bc3568d5-6a63-43db-8cf6-998f232cd0d6","Type":"ContainerStarted","Data":"d1e97736cb393f44bc6b7538cdff396e704c756078dd6cb8d708063e506d9f59"}
Nov 03 00:15:21.004069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:21.004031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:21.108835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:21.108803    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:22.182012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:22.181974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:22.299027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:22.298993    4550 generic.go:356] "Generic (PLEG): container finished" podID="37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" containerID="517ec9c41397a0c2a0cdc961e408d415ee8226908fbee9767c1d3e21eb81e268" exitCode=0
Nov 03 00:15:22.299211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:22.299076    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c" event={"ID":"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3","Type":"ContainerDied","Data":"517ec9c41397a0c2a0cdc961e408d415ee8226908fbee9767c1d3e21eb81e268"}
Nov 03 00:15:22.300751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:22.300721    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4" event={"ID":"bc3568d5-6a63-43db-8cf6-998f232cd0d6","Type":"ContainerStarted","Data":"23cfb6e60135b8dac98a49d53b97f2dc5aa33313666c6816cd63a2a877e1ea6a"}
Nov 03 00:15:22.394257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:22.394147    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4" podStartSLOduration=2.189673208 podStartE2EDuration="5.394131738s" totalImagesPullingTime="3.20445853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:22.392717962 +0000 UTC m=+604.714254761" watchObservedRunningTime="2026-11-03 00:15:22.394131738 +0000 UTC m=+604.715668538"
Nov 03 00:15:23.049753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.049664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:23.307023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.306936    4550 generic.go:356] "Generic (PLEG): container finished" podID="bc3568d5-6a63-43db-8cf6-998f232cd0d6" containerID="23cfb6e60135b8dac98a49d53b97f2dc5aa33313666c6816cd63a2a877e1ea6a" exitCode=0
Nov 03 00:15:23.307411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.307022    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4" event={"ID":"bc3568d5-6a63-43db-8cf6-998f232cd0d6","Type":"ContainerDied","Data":"23cfb6e60135b8dac98a49d53b97f2dc5aa33313666c6816cd63a2a877e1ea6a"}
Nov 03 00:15:23.459667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.459638    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:23.476975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.476943    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca\") pod \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") "
Nov 03 00:15:23.477139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.477004    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca\") pod \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") "
Nov 03 00:15:23.477139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.477038    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq\" (UniqueName: \"kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq\") pod \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\" (UID: \"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3\") "
Nov 03 00:15:23.477384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.477347    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca" pod "37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" (UID: "37d99ed2-1f12-4bb9-a672-908fcf0a9ab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:23.479268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.479245    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca" pod "37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" (UID: "37d99ed2-1f12-4bb9-a672-908fcf0a9ab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:23.479349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.479248    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq" pod "37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" (UID: "37d99ed2-1f12-4bb9-a672-908fcf0a9ab3"). InnerVolumeSpecName "kube-api-access-pkfkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:23.577840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.577763    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:23.577840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.577791    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:23.577840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:23.577803    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfkq\" (UniqueName: \"kubernetes.io/projected/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3-kube-api-access-pkfkq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:24.312524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.312490    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"
Nov 03 00:15:24.312524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.312500    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c" event={"ID":"37d99ed2-1f12-4bb9-a672-908fcf0a9ab3","Type":"ContainerDied","Data":"ae2498c0b2c98073b4265a2e83f1b8a731a28b5c05c7c331fc5bd21d28b38ecd"}
Nov 03 00:15:24.313101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.312560    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae2498c0b2c98073b4265a2e83f1b8a731a28b5c05c7c331fc5bd21d28b38ecd"
Nov 03 00:15:24.459887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.459861    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:24.485103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.485068    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca\") pod \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") "
Nov 03 00:15:24.485292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.485121    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca\") pod \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") "
Nov 03 00:15:24.485292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.485187    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn\" (UniqueName: \"kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn\") pod \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\" (UID: \"bc3568d5-6a63-43db-8cf6-998f232cd0d6\") "
Nov 03 00:15:24.485569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.485535    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca" pod "bc3568d5-6a63-43db-8cf6-998f232cd0d6" (UID: "bc3568d5-6a63-43db-8cf6-998f232cd0d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:24.487707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.487678    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca" pod "bc3568d5-6a63-43db-8cf6-998f232cd0d6" (UID: "bc3568d5-6a63-43db-8cf6-998f232cd0d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:24.487950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.487924    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn" pod "bc3568d5-6a63-43db-8cf6-998f232cd0d6" (UID: "bc3568d5-6a63-43db-8cf6-998f232cd0d6"). InnerVolumeSpecName "kube-api-access-pnstn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:24.586109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.586017    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:24.586109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.586050    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc3568d5-6a63-43db-8cf6-998f232cd0d6-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:24.586109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.586060    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnstn\" (UniqueName: \"kubernetes.io/projected/bc3568d5-6a63-43db-8cf6-998f232cd0d6-kube-api-access-pnstn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:24.725132 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:15:24.725088    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:15:24.731250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731223    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"]
Nov 03 00:15:24.731611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731587    4550 cpu_manager.go:414] "containerMap: removing container" podUID="bc3568d5-6a63-43db-8cf6-998f232cd0d6" containerName="revoke-keys"
Nov 03 00:15:24.731611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731603    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc3568d5-6a63-43db-8cf6-998f232cd0d6" containerName="revoke-keys"
Nov 03 00:15:24.731757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731627    4550 cpu_manager.go:414] "containerMap: removing container" podUID="37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" containerName="revoke-keys"
Nov 03 00:15:24.731757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731632    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" containerName="revoke-keys"
Nov 03 00:15:24.731757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731691    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" containerName="revoke-keys"
Nov 03 00:15:24.731757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.731701    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc3568d5-6a63-43db-8cf6-998f232cd0d6" containerName="revoke-keys"
Nov 03 00:15:24.734898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.734880    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.754016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.753991    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"]
Nov 03 00:15:24.787252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.787208    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwd2\" (UniqueName: \"kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.787455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.787357    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.787532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.787471    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.841378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.841289    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:24.888237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.888190    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.888458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.888331    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwd2\" (UniqueName: \"kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.888458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.888411    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.889129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.889069    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.891036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.891007    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:24.909588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:24.909551    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwd2\" (UniqueName: \"kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2\") pod \"maas-api-revoke-keys-e2e-ait-19590f89-mp66b\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:25.049161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:25.049127    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:25.319552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:25.319520    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"
Nov 03 00:15:25.319552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:25.319537    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4" event={"ID":"bc3568d5-6a63-43db-8cf6-998f232cd0d6","Type":"ContainerDied","Data":"d1e97736cb393f44bc6b7538cdff396e704c756078dd6cb8d708063e506d9f59"}
Nov 03 00:15:25.320047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:25.319585    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1e97736cb393f44bc6b7538cdff396e704c756078dd6cb8d708063e506d9f59"
Nov 03 00:15:25.343865 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:15:25.343834    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473805e6_b4ec_4218_8b78_2fa6520bd2a1.slice/crio-37d08c7dabcf0399cdf6a7a5a23d36416d56a5eae0cc89f86d5d8f452d73ec4d WatchSource:0}: Error finding container 37d08c7dabcf0399cdf6a7a5a23d36416d56a5eae0cc89f86d5d8f452d73ec4d: Status 404 returned error can't find the container with id 37d08c7dabcf0399cdf6a7a5a23d36416d56a5eae0cc89f86d5d8f452d73ec4d
Nov 03 00:15:25.344269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:25.344243    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"]
Nov 03 00:15:26.326204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:26.326169    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b" event={"ID":"473805e6-b4ec-4218-8b78-2fa6520bd2a1","Type":"ContainerStarted","Data":"717327e6bd111647c1bffc3892f660ab711258e5e92771b63218598e35a5eebe"}
Nov 03 00:15:26.326624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:26.326227    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b" event={"ID":"473805e6-b4ec-4218-8b78-2fa6520bd2a1","Type":"ContainerStarted","Data":"37d08c7dabcf0399cdf6a7a5a23d36416d56a5eae0cc89f86d5d8f452d73ec4d"}
Nov 03 00:15:26.371102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:26.371050    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b" podStartSLOduration=2.371035264 podStartE2EDuration="2.371035264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:26.369848017 +0000 UTC m=+608.691384819" watchObservedRunningTime="2026-11-03 00:15:26.371035264 +0000 UTC m=+608.692572118"
Nov 03 00:15:26.736281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:26.736242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:27.332514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:27.332472    4550 generic.go:356] "Generic (PLEG): container finished" podID="473805e6-b4ec-4218-8b78-2fa6520bd2a1" containerID="717327e6bd111647c1bffc3892f660ab711258e5e92771b63218598e35a5eebe" exitCode=0
Nov 03 00:15:27.332992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:27.332545    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b" event={"ID":"473805e6-b4ec-4218-8b78-2fa6520bd2a1","Type":"ContainerDied","Data":"717327e6bd111647c1bffc3892f660ab711258e5e92771b63218598e35a5eebe"}
Nov 03 00:15:28.477468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.477425    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:28.519006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.518974    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca\") pod \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") "
Nov 03 00:15:28.519176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.519069    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca\") pod \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") "
Nov 03 00:15:28.519176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.519106    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2\" (UniqueName: \"kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2\") pod \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\" (UID: \"473805e6-b4ec-4218-8b78-2fa6520bd2a1\") "
Nov 03 00:15:28.519381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.519354    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca" pod "473805e6-b4ec-4218-8b78-2fa6520bd2a1" (UID: "473805e6-b4ec-4218-8b78-2fa6520bd2a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:28.521515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.521489    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2" pod "473805e6-b4ec-4218-8b78-2fa6520bd2a1" (UID: "473805e6-b4ec-4218-8b78-2fa6520bd2a1"). InnerVolumeSpecName "kube-api-access-9zwd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:28.521662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.521583    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca" pod "473805e6-b4ec-4218-8b78-2fa6520bd2a1" (UID: "473805e6-b4ec-4218-8b78-2fa6520bd2a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:28.602972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.602865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:28.620179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.620145    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:28.620179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.620180    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/473805e6-b4ec-4218-8b78-2fa6520bd2a1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:28.620375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.620195    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zwd2\" (UniqueName: \"kubernetes.io/projected/473805e6-b4ec-4218-8b78-2fa6520bd2a1-kube-api-access-9zwd2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:28.651216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.651182    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"]
Nov 03 00:15:28.651590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.651576    4550 cpu_manager.go:414] "containerMap: removing container" podUID="473805e6-b4ec-4218-8b78-2fa6520bd2a1" containerName="revoke-keys"
Nov 03 00:15:28.651638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.651591    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="473805e6-b4ec-4218-8b78-2fa6520bd2a1" containerName="revoke-keys"
Nov 03 00:15:28.651680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.651670    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="473805e6-b4ec-4218-8b78-2fa6520bd2a1" containerName="revoke-keys"
Nov 03 00:15:28.654821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.654794    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.675947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.675920    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"]
Nov 03 00:15:28.702269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.702237    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:28.721303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.721276    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.721303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.721305    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.721552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.721451    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhbt\" (UniqueName: \"kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.822039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.822007    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhbt\" (UniqueName: \"kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.822301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.822150    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.822301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.822184    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.822922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.822897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.824729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.824711    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.842764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.842741    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhbt\" (UniqueName: \"kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt\") pod \"maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:28.971404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:28.971364    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:29.263472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:29.263421    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"]
Nov 03 00:15:29.263807 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:15:29.263782    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585b7fc8_38db_45c8_80e9_9a839825c9c6.slice/crio-e98da200b32856b37af1a2ff8bcad2d51fb136f09cd089573e58bb515c2409cc WatchSource:0}: Error finding container e98da200b32856b37af1a2ff8bcad2d51fb136f09cd089573e58bb515c2409cc: Status 404 returned error can't find the container with id e98da200b32856b37af1a2ff8bcad2d51fb136f09cd089573e58bb515c2409cc
Nov 03 00:15:29.343836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:29.343806    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"
Nov 03 00:15:29.344003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:29.343823    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b" event={"ID":"473805e6-b4ec-4218-8b78-2fa6520bd2a1","Type":"ContainerDied","Data":"37d08c7dabcf0399cdf6a7a5a23d36416d56a5eae0cc89f86d5d8f452d73ec4d"}
Nov 03 00:15:29.344003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:29.343903    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d08c7dabcf0399cdf6a7a5a23d36416d56a5eae0cc89f86d5d8f452d73ec4d"
Nov 03 00:15:29.345202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:29.345179    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc" event={"ID":"585b7fc8-38db-45c8-80e9-9a839825c9c6","Type":"ContainerStarted","Data":"e98da200b32856b37af1a2ff8bcad2d51fb136f09cd089573e58bb515c2409cc"}
Nov 03 00:15:29.927741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:29.927694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:30.027821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:30.027787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:30.351407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:30.351321    4550 generic.go:356] "Generic (PLEG): container finished" podID="585b7fc8-38db-45c8-80e9-9a839825c9c6" containerID="af21a536ba4bd1187ae1ac239d04fe3d7ccb00e5b8f56a4891e46ce6bdd1289d" exitCode=0
Nov 03 00:15:30.351407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:30.351364    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc" event={"ID":"585b7fc8-38db-45c8-80e9-9a839825c9c6","Type":"ContainerDied","Data":"af21a536ba4bd1187ae1ac239d04fe3d7ccb00e5b8f56a4891e46ce6bdd1289d"}
Nov 03 00:15:30.688084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:30.688049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:31.495364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.495334    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:31.548352 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.548320    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca\") pod \"585b7fc8-38db-45c8-80e9-9a839825c9c6\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") "
Nov 03 00:15:31.548558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.548361    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt\" (UniqueName: \"kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt\") pod \"585b7fc8-38db-45c8-80e9-9a839825c9c6\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") "
Nov 03 00:15:31.548558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.548402    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca\") pod \"585b7fc8-38db-45c8-80e9-9a839825c9c6\" (UID: \"585b7fc8-38db-45c8-80e9-9a839825c9c6\") "
Nov 03 00:15:31.548837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.548801    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca" pod "585b7fc8-38db-45c8-80e9-9a839825c9c6" (UID: "585b7fc8-38db-45c8-80e9-9a839825c9c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:31.550772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.550729    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt" pod "585b7fc8-38db-45c8-80e9-9a839825c9c6" (UID: "585b7fc8-38db-45c8-80e9-9a839825c9c6"). InnerVolumeSpecName "kube-api-access-5qhbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:31.551008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.550985    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca" pod "585b7fc8-38db-45c8-80e9-9a839825c9c6" (UID: "585b7fc8-38db-45c8-80e9-9a839825c9c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:31.650027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.649996    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:31.650027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.650027    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qhbt\" (UniqueName: \"kubernetes.io/projected/585b7fc8-38db-45c8-80e9-9a839825c9c6-kube-api-access-5qhbt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:31.650228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:31.650037    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585b7fc8-38db-45c8-80e9-9a839825c9c6-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:15:32.362410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:32.362375    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"
Nov 03 00:15:32.362608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:32.362406    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc" event={"ID":"585b7fc8-38db-45c8-80e9-9a839825c9c6","Type":"ContainerDied","Data":"e98da200b32856b37af1a2ff8bcad2d51fb136f09cd089573e58bb515c2409cc"}
Nov 03 00:15:32.362608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:32.362473    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98da200b32856b37af1a2ff8bcad2d51fb136f09cd089573e58bb515c2409cc"
Nov 03 00:15:32.448167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:32.448132    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:33.916060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:33.916018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:34.496987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:34.496952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:34.664336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:34.664299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:34.713603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:34.713564    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"]
Nov 03 00:15:34.726955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:34.726927    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c600f063-wlqr4"]
Nov 03 00:15:34.753743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:34.753656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:35.397483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:35.397448    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:35.497742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:35.497699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:35.806135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:35.806092    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc3568d5-6a63-43db-8cf6-998f232cd0d6" path="/var/lib/kubelet/pods/bc3568d5-6a63-43db-8cf6-998f232cd0d6/volumes"
Nov 03 00:15:36.282900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:36.282592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:36.396750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:36.396713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:36.817845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:36.817811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:36.918330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:36.918294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:36.956245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:36.956213    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"]
Nov 03 00:15:36.968777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:36.968748    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b4cc5943-l6l5c"]
Nov 03 00:15:37.018926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.018890    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:37.110493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.110391    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:37.193978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.193936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:37.518080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.518044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:37.793744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.793652    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:37.806941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.806911    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d99ed2-1f12-4bb9-a672-908fcf0a9ab3" path="/var/lib/kubelet/pods/37d99ed2-1f12-4bb9-a672-908fcf0a9ab3/volumes"
Nov 03 00:15:37.912736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:37.912696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:38.416321 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:38.416282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:38.611752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:38.611718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:38.717984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:38.717894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:39.136008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.135892    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:39.241405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.241365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:39.271885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.271855    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"]
Nov 03 00:15:39.284876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.284844    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19590f89-mp66b"]
Nov 03 00:15:39.343017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.342974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:39.806050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.806005    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="473805e6-b4ec-4218-8b78-2fa6520bd2a1" path="/var/lib/kubelet/pods/473805e6-b4ec-4218-8b78-2fa6520bd2a1/volumes"
Nov 03 00:15:39.978740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:39.978709    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:40.733057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:40.733010    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:40.886350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:40.886106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.262327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.262288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.378764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.378727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.473607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.473566    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.574580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.574474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.691348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.691308    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.791020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.790987    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:41.840334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.840244    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"]
Nov 03 00:15:41.852679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:41.852648    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a96a129b-ffpzc"]
Nov 03 00:15:42.051896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:42.051857    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:42.316273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:42.316239    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:42.426505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:42.426466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:42.810166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:42.810123    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:43.039408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:43.039370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:43.149914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:43.149884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:43.337199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:43.337151    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:43.559606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:43.559515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:43.807345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:43.807304    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585b7fc8-38db-45c8-80e9-9a839825c9c6" path="/var/lib/kubelet/pods/585b7fc8-38db-45c8-80e9-9a839825c9c6/volumes"
Nov 03 00:15:48.758313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:48.758273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:48.858818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:48.858782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:49.775936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:49.775899    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:50.024650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:50.024613    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:50.122412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:50.122326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:51.665286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:51.665244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:51.753381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:51.753346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:51.846323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:51.846285    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:53.254036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:53.253999    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:53.790557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:53.790522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:54.028418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:54.028373    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:54.174329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:54.174291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:54.289193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:54.289157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:54.725206 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:15:54.725156    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:15:54.970011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:54.969971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:55.490269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:55.490231    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:55.603565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:55.603524    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:56.061151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:56.061114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:57.973583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:57.973536    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:59.456026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:59.455990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:59.608361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:59.608328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:15:59.963681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:15:59.963646    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:00.600976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:00.600938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:01.234844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:01.234811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:01.354269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:01.354214    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:01.742832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:01.742800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:01.917944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:01.917907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:02.076453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:02.076349    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:02.193596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:02.193553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:02.326842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:02.326733    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:02.903320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:02.903281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:03.039003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:03.038967    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:03.241294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:03.241214    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:04.359570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:04.359519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:04.778958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:04.778921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:05.427029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.426992    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:05.434836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.434807    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"]
Nov 03 00:16:05.435185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.435169    4550 cpu_manager.go:414] "containerMap: removing container" podUID="585b7fc8-38db-45c8-80e9-9a839825c9c6" containerName="revoke-keys"
Nov 03 00:16:05.435185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.435183    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585b7fc8-38db-45c8-80e9-9a839825c9c6" containerName="revoke-keys"
Nov 03 00:16:05.435331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.435264    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="585b7fc8-38db-45c8-80e9-9a839825c9c6" containerName="revoke-keys"
Nov 03 00:16:05.438126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.438105    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.473621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.473592    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"kube-root-ca.crt\""
Nov 03 00:16:05.473782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.473593    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"openshift-service-ca.crt\""
Nov 03 00:16:05.488250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.488189    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"simulator-w4-99bdb8-kserve-self-signed-certs\""
Nov 03 00:16:05.488250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.488181    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"default-dockercfg-7knb7\""
Nov 03 00:16:05.488986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.488943    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.489106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.489006    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.489106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.489025    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.489226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.489145    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmcw\" (UniqueName: \"kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.489226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.489200    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.489336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.489247    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.490652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.490627    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"]
Nov 03 00:16:05.590316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590271    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590313    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590336    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590381    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmcw\" (UniqueName: \"kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590419    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590477    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590835    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590879    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.590978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.590916    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.592737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.592709    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.592899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.592881    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.617457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.617394    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmcw\" (UniqueName: \"kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw\") pod \"simulator-w4-99bdb8-kserve-8494847845-mxsfz\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") " pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:05.640836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.640805    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:05.749869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:05.749784    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:06.073002 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:16:06.072973    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a89531_cdc1_4e23_bf99_3235a955c60f.slice/crio-320e4769083fba633a31f982b547559f90b7c9b551668a7e205c348b621859de WatchSource:0}: Error finding container 320e4769083fba633a31f982b547559f90b7c9b551668a7e205c348b621859de: Status 404 returned error can't find the container with id 320e4769083fba633a31f982b547559f90b7c9b551668a7e205c348b621859de
Nov 03 00:16:06.073489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:06.073410    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"]
Nov 03 00:16:06.075083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:06.075067    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:16:06.459503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:06.459417    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:06.525666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:06.525603    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" event={"ID":"c8a89531-cdc1-4e23-bf99-3235a955c60f","Type":"ContainerStarted","Data":"320e4769083fba633a31f982b547559f90b7c9b551668a7e205c348b621859de"}
Nov 03 00:16:07.533061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:07.533023    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" event={"ID":"c8a89531-cdc1-4e23-bf99-3235a955c60f","Type":"ContainerStarted","Data":"285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128"}
Nov 03 00:16:07.577974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:07.577921    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podStartSLOduration=1.315359491 podStartE2EDuration="2.577905424s" totalImagesPullingTime="1.262545933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:07.576371855 +0000 UTC m=+649.897908669" watchObservedRunningTime="2026-11-03 00:16:07.577905424 +0000 UTC m=+649.899442224"
Nov 03 00:16:07.750401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:07.750368    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:08.246149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:08.246118    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:08.526941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:08.526852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:09.970962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:09.970929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:10.064324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:10.064294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:10.164902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:10.164863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:11.678652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:11.678610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:11.955662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:11.955558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:12.329943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:12.329859    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:12.441007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:12.440964    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:12.589405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:12.589320    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:12.704375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:12.704341    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:13.753916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:13.753882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:14.858501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:14.858464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:14.972805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:14.972765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:15.412124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:15.412092    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:15.534795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:15.534759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:15.751054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:15.750976    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:15.769765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:15.769738    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:16.402147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.402110    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"]
Nov 03 00:16:16.405661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.405635    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.437508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.437481    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"kube-root-ca.crt\""
Nov 03 00:16:16.437508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.437481    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"openshift-service-ca.crt\""
Nov 03 00:16:16.440046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.440022    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"]
Nov 03 00:16:16.441809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.441791    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"default-dockercfg-lqdnc\""
Nov 03 00:16:16.441901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.441791    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"simulator-w2-4c61c8-kserve-self-signed-certs\""
Nov 03 00:16:16.449424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.449402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:16.498230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.498190    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.498230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.498229    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.498508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.498252    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.498508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.498402    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.498508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.498465    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.498508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.498502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv45n\" (UniqueName: \"kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.584973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.584938    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:16:16.599458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599417    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599480    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv45n\" (UniqueName: \"kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599600    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599626    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599666    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599826    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.599942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599926    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.600041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.599965    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.601975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.601948    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.602225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.602199    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.633723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.633658    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv45n\" (UniqueName: \"kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n\") pod \"simulator-w2-4c61c8-kserve-76479cf6bd-glkbv\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") " pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:16.720207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:16.720129    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:17.023858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:17.023797    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"]
Nov 03 00:16:17.024308 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:16:17.024259    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9e2d0ee_8a45_4468_afbb_c49487002bb3.slice/crio-f9493d19af8347d57c45dde39b3d5f9ee174528c11bff2e4005657627f410459 WatchSource:0}: Error finding container f9493d19af8347d57c45dde39b3d5f9ee174528c11bff2e4005657627f410459: Status 404 returned error can't find the container with id f9493d19af8347d57c45dde39b3d5f9ee174528c11bff2e4005657627f410459
Nov 03 00:16:17.239793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:17.239756    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:17.580232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:17.580196    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" event={"ID":"c9e2d0ee-8a45-4468-afbb-c49487002bb3","Type":"ContainerStarted","Data":"8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39"}
Nov 03 00:16:17.580673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:17.580247    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" event={"ID":"c9e2d0ee-8a45-4468-afbb-c49487002bb3","Type":"ContainerStarted","Data":"f9493d19af8347d57c45dde39b3d5f9ee174528c11bff2e4005657627f410459"}
Nov 03 00:16:17.624074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:17.624018    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podStartSLOduration=1.624004066 podStartE2EDuration="1.624004066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:17.622955789 +0000 UTC m=+659.944492612" watchObservedRunningTime="2026-11-03 00:16:17.624004066 +0000 UTC m=+659.945540866"
Nov 03 00:16:17.720993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:17.720957    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:19.047161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:19.047100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:20.775994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:20.775952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:22.432236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:22.432192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:24.286607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.286566    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:24.605518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.605412    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:24.724904 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:16:24.724860    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:16:24.741959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.741926    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"]
Nov 03 00:16:24.746881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.746856    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.767111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.767081    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"]
Nov 03 00:16:24.777794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.777760    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"kube-root-ca.crt\""
Nov 03 00:16:24.777794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.777775    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"openshift-service-ca.crt\""
Nov 03 00:16:24.778001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.777888    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"simulator-w0-fc9316-kserve-self-signed-certs\""
Nov 03 00:16:24.778653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.778632    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"default-dockercfg-qffjg\""
Nov 03 00:16:24.790678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.790641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:24.880067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.879981    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w48v\" (UniqueName: \"kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.880233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.880099    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.880233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.880129    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.880233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.880201    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.880495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.880453    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.880582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.880513    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.981574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.981539    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.981768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.981594    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.981768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.981680    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.981768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.981703    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.981768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.981758    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w48v\" (UniqueName: \"kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.982003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.981818    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.982083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.982050    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.982146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.982097    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.982202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.982140    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.984290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.984262    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:24.984508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:24.984488    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:25.003688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:25.003655    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w48v\" (UniqueName: \"kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v\") pod \"simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") " pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:25.061413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:25.061372    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:25.373813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:25.373783    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"]
Nov 03 00:16:25.374282 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:16:25.374257    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda20455af_f5bf_41ca_a942_8c2b6b35d962.slice/crio-71fe5109fd31a1fadd88c440689a5d6779cbfb138d803cdff27dd3435a90df09 WatchSource:0}: Error finding container 71fe5109fd31a1fadd88c440689a5d6779cbfb138d803cdff27dd3435a90df09: Status 404 returned error can't find the container with id 71fe5109fd31a1fadd88c440689a5d6779cbfb138d803cdff27dd3435a90df09
Nov 03 00:16:25.622303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:25.622194    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" event={"ID":"a20455af-f5bf-41ca-a942-8c2b6b35d962","Type":"ContainerStarted","Data":"5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83"}
Nov 03 00:16:25.622303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:25.622260    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" event={"ID":"a20455af-f5bf-41ca-a942-8c2b6b35d962","Type":"ContainerStarted","Data":"71fe5109fd31a1fadd88c440689a5d6779cbfb138d803cdff27dd3435a90df09"}
Nov 03 00:16:25.667132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:25.667069    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podStartSLOduration=1.66705393 podStartE2EDuration="1.66705393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:25.665685973 +0000 UTC m=+667.987222766" watchObservedRunningTime="2026-11-03 00:16:25.66705393 +0000 UTC m=+667.988590721"
Nov 03 00:16:26.038757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:26.038710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:26.062467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:26.062410    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:26.720952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:26.720917    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:26.734185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:26.734152    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:27.624775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:27.624730    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:27.644553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:27.644522    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:16:29.445973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:29.445887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:31.139754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:31.139710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:32.829880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:32.829847    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:34.434699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:34.434660    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:35.062233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:35.062183    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:35.075129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:35.075099    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:35.682567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:35.682527    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:16:36.058355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:36.058257    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:37.621741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:37.621703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:39.215902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:39.215856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:40.912315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:40.912280    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:42.614578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:42.614539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:44.217811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:44.217776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:45.826811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:45.826770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:47.423880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:47.423844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:49.009683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:49.009645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:50.886991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:50.886953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:51.017547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:51.017509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:51.134293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:51.134256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:52.999140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:52.999106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:54.723573 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:16:54.723536    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:16:54.847686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:54.847642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:56.001827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.001793    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"]
Nov 03 00:16:56.006849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.006826    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.042760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.042725    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"]
Nov 03 00:16:56.053212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.053179    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"kube-root-ca.crt\""
Nov 03 00:16:56.053403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.053211    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"openshift-service-ca.crt\""
Nov 03 00:16:56.053403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.053353    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"simulator-w1-b85343-kserve-self-signed-certs\""
Nov 03 00:16:56.053912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.053896    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"default-dockercfg-hzjrw\""
Nov 03 00:16:56.060489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.060468    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.060576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.060500    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.060576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.060534    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b49jg\" (UniqueName: \"kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.060681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.060663    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.060730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.060717    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.060776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.060740    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.067867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.067826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:56.161820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.161777    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.161827    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.161874    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b49jg\" (UniqueName: \"kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.161899    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.161962    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.161987    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.162286    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.162297    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.162598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.162573    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.164416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.164389    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.164664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.164649    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.184894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.184860    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b49jg\" (UniqueName: \"kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg\") pod \"simulator-w1-b85343-kserve-5f875cc989-98rt8\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") " pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.319269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.319188    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:56.625587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.625551    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"]
Nov 03 00:16:56.625833 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:16:56.625798    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8919c830_d611_47e6_91c9_54e3d764d90e.slice/crio-b24e316df3791e7f51c77246dab1df0ce4cb94382813e776cf8019cbb0647f82 WatchSource:0}: Error finding container b24e316df3791e7f51c77246dab1df0ce4cb94382813e776cf8019cbb0647f82: Status 404 returned error can't find the container with id b24e316df3791e7f51c77246dab1df0ce4cb94382813e776cf8019cbb0647f82
Nov 03 00:16:56.754195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.754162    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:16:56.773310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.773269    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" event={"ID":"8919c830-d611-47e6-91c9-54e3d764d90e","Type":"ContainerStarted","Data":"538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84"}
Nov 03 00:16:56.773310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.773312    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" event={"ID":"8919c830-d611-47e6-91c9-54e3d764d90e","Type":"ContainerStarted","Data":"b24e316df3791e7f51c77246dab1df0ce4cb94382813e776cf8019cbb0647f82"}
Nov 03 00:16:56.814894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:56.814828    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podStartSLOduration=1.814809318 podStartE2EDuration="1.814809318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:56.812410454 +0000 UTC m=+699.133947273" watchObservedRunningTime="2026-11-03 00:16:56.814809318 +0000 UTC m=+699.136346118"
Nov 03 00:16:57.320035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:57.320001    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:16:58.589410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:16:58.589378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:00.230328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:00.230288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:01.816444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:01.816404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:03.513019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:03.512983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:04.919122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:04.919082    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:05.310796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:05.310706    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:06.320275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:06.320234    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:17:06.334469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:06.334408    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:17:06.833318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:06.833287    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:17:06.915825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:06.915791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:08.625468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.625415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:08.699371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.699338    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"]
Nov 03 00:17:08.703442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.703405    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.721468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.721426    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"]
Nov 03 00:17:08.722314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.722270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:08.732680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.732648    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"premium-w4-99bdb8-kserve-self-signed-certs\""
Nov 03 00:17:08.781655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.781618    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.781655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.781666    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.781876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.781715    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.781876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.781828    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.781944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.781901    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.781944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.781939    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7dnv\" (UniqueName: \"kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.816126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.816090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:08.883387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.883287    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7dnv\" (UniqueName: \"kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.883387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.883348    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.883649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.883389    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.883649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.883462    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.883649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.883488    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.883649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.883543    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.884090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.884057    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.884602 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.884578    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.885516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.885476    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.887930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.887906    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.888332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.888314    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:08.903415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:08.903387    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7dnv\" (UniqueName: \"kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv\") pod \"premium-w4-99bdb8-kserve-54cdd7798d-cl4kj\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") " pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:09.020144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:09.020109    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:09.318960 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:17:09.318930    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17326228_95bb_449c_ac56_919975c8b94d.slice/crio-9274d459ffec47f150e0d3d72a5129b6e8eb3d654f83b42b9bf3085be0751945 WatchSource:0}: Error finding container 9274d459ffec47f150e0d3d72a5129b6e8eb3d654f83b42b9bf3085be0751945: Status 404 returned error can't find the container with id 9274d459ffec47f150e0d3d72a5129b6e8eb3d654f83b42b9bf3085be0751945
Nov 03 00:17:09.319124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:09.319103    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"]
Nov 03 00:17:09.838507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:09.838468    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" event={"ID":"17326228-95bb-449c-ac56-919975c8b94d","Type":"ContainerStarted","Data":"0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1"}
Nov 03 00:17:09.838918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:09.838518    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" event={"ID":"17326228-95bb-449c-ac56-919975c8b94d","Type":"ContainerStarted","Data":"9274d459ffec47f150e0d3d72a5129b6e8eb3d654f83b42b9bf3085be0751945"}
Nov 03 00:17:09.880032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:09.879967    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podStartSLOduration=1.8799509859999999 podStartE2EDuration="1.879950986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:09.878482683 +0000 UTC m=+712.200019529" watchObservedRunningTime="2026-11-03 00:17:09.879950986 +0000 UTC m=+712.201487785"
Nov 03 00:17:10.021095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:10.021058    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:10.325358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:10.325314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:11.441036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:11.440994    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:12.013759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:12.013717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:13.478561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.478524    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"]
Nov 03 00:17:13.483366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.483337    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.515394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.515366    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"openshift-service-ca.crt\""
Nov 03 00:17:13.515626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.515594    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"kube-root-ca.crt\""
Nov 03 00:17:13.515626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.515603    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"premium-w3-8e90c5-kserve-self-signed-certs\""
Nov 03 00:17:13.515811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.515644    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"default-dockercfg-79f5r\""
Nov 03 00:17:13.517393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.517368    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"]
Nov 03 00:17:13.533932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.533900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:13.629301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.629259    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftxfd\" (UniqueName: \"kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.629513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.629316    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.629513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.629478    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.629599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.629518    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.629671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.629647    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.629726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.629708    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.730942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.730854    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.730942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.730913    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.730973    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.731018    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.731078    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftxfd\" (UniqueName: \"kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.731113    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.731449    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.731672    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.731893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.731871    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.733358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.733338    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.733672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.733650    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.752626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.752597    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftxfd\" (UniqueName: \"kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd\") pod \"premium-w3-8e90c5-kserve-7576f576df-zwrj6\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") " pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:13.794076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.794038    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:13.798830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:13.798801    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:14.099266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:14.099220    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"]
Nov 03 00:17:14.099394 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:17:14.099233    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode697d77c_8549_49d9_b5c1_2f68e6ff53f3.slice/crio-70f62909058d7af4bf0740ad55e9a8130928dbeecc4d599487333bece6c67830 WatchSource:0}: Error finding container 70f62909058d7af4bf0740ad55e9a8130928dbeecc4d599487333bece6c67830: Status 404 returned error can't find the container with id 70f62909058d7af4bf0740ad55e9a8130928dbeecc4d599487333bece6c67830
Nov 03 00:17:14.870720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:14.870678    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" event={"ID":"e697d77c-8549-49d9-b5c1-2f68e6ff53f3","Type":"ContainerStarted","Data":"b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff"}
Nov 03 00:17:14.871178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:14.870731    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" event={"ID":"e697d77c-8549-49d9-b5c1-2f68e6ff53f3","Type":"ContainerStarted","Data":"70f62909058d7af4bf0740ad55e9a8130928dbeecc4d599487333bece6c67830"}
Nov 03 00:17:14.914703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:14.914639    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podStartSLOduration=1.9146211210000001 podStartE2EDuration="1.914621121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:14.91420437 +0000 UTC m=+717.235741170" watchObservedRunningTime="2026-11-03 00:17:14.914621121 +0000 UTC m=+717.236157923"
Nov 03 00:17:15.535708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:15.535671    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:15.799809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:15.799724    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:17.218755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:17.218720    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:18.812451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:18.812402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:19.021248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:19.021209    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:19.034277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:19.034245    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:19.908999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:19.908973    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:17:20.424143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:20.424110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:22.009657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:22.009620    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:23.727815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:23.727776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:23.799222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:23.799190    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:23.813631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:23.813607    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:23.936021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:23.935988    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:17:24.727327 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:17:24.727294    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:17:25.431460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:25.431400    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:27.110560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:27.110522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:27.718784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:27.718741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:28.813114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:28.813078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:30.499731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:30.499686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:32.148448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:32.148395    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:33.928083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:33.928046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:35.642363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:35.642332    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:37.359219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:37.359184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:39.269714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:39.269670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:41.026573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:41.026535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:42.622954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:42.622916    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:44.336853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:44.336822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:46.228687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:46.228648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:47.969183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:47.969136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:49.552755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:49.552711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:51.113987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:51.113949    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:52.822037 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:52.821997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:54.526318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:54.526282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:54.722397 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:17:54.722363    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:17:56.321944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:56.321910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:57.303937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.303901    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"]
Nov 03 00:17:57.307714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.307696    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.323643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.323611    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:17:57.336796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.336769    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:17:57.338468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.338444    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"]
Nov 03 00:17:57.465597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.465558    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.465597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.465605    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgd5f\" (UniqueName: \"kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.465810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.465626    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.567266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.567188    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.567266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.567229    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgd5f\" (UniqueName: \"kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.567266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.567263    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.567979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.567958    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.569570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.569546    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.588140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.588111    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgd5f\" (UniqueName: \"kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f\") pod \"maas-api-revoke-sub-9bcf952c-wpw6t\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.621104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.621074    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:17:57.920085 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:17:57.920046    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3940b3_bdf1_4ca2_9955_fd60a8d74340.slice/crio-58d369beb32a02e88115dbd5164f70e616f54a256fd8ebdb1d3f1f57461e6360 WatchSource:0}: Error finding container 58d369beb32a02e88115dbd5164f70e616f54a256fd8ebdb1d3f1f57461e6360: Status 404 returned error can't find the container with id 58d369beb32a02e88115dbd5164f70e616f54a256fd8ebdb1d3f1f57461e6360
Nov 03 00:17:57.920853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.920823    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"]
Nov 03 00:17:57.943648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:57.943617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:17:58.094414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:58.094375    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t" event={"ID":"dc3940b3-bdf1-4ca2-9955-fd60a8d74340","Type":"ContainerStarted","Data":"58d369beb32a02e88115dbd5164f70e616f54a256fd8ebdb1d3f1f57461e6360"}
Nov 03 00:17:59.101224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:59.101132    4550 generic.go:356] "Generic (PLEG): container finished" podID="dc3940b3-bdf1-4ca2-9955-fd60a8d74340" containerID="932041b2e164740a9669a13c1b5f59f36da0e85c2d830a091a486865d955bb30" exitCode=0
Nov 03 00:17:59.101224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:59.101196    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t" event={"ID":"dc3940b3-bdf1-4ca2-9955-fd60a8d74340","Type":"ContainerDied","Data":"932041b2e164740a9669a13c1b5f59f36da0e85c2d830a091a486865d955bb30"}
Nov 03 00:17:59.527763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:17:59.527670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:00.260521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.260485    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:18:00.398698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.398619    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f\" (UniqueName: \"kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f\") pod \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") "
Nov 03 00:18:00.398880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.398724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca\") pod \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") "
Nov 03 00:18:00.398880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.398788    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca\") pod \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\" (UID: \"dc3940b3-bdf1-4ca2-9955-fd60a8d74340\") "
Nov 03 00:18:00.399235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.399199    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca" pod "dc3940b3-bdf1-4ca2-9955-fd60a8d74340" (UID: "dc3940b3-bdf1-4ca2-9955-fd60a8d74340"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:00.400852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.400824    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca" pod "dc3940b3-bdf1-4ca2-9955-fd60a8d74340" (UID: "dc3940b3-bdf1-4ca2-9955-fd60a8d74340"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:00.400971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.400946    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f" pod "dc3940b3-bdf1-4ca2-9955-fd60a8d74340" (UID: "dc3940b3-bdf1-4ca2-9955-fd60a8d74340"). InnerVolumeSpecName "kube-api-access-qgd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:00.500422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.500380    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:00.500619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.500424    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:00.500619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:00.500471    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgd5f\" (UniqueName: \"kubernetes.io/projected/dc3940b3-bdf1-4ca2-9955-fd60a8d74340-kube-api-access-qgd5f\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:01.114104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:01.114062    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t" event={"ID":"dc3940b3-bdf1-4ca2-9955-fd60a8d74340","Type":"ContainerDied","Data":"58d369beb32a02e88115dbd5164f70e616f54a256fd8ebdb1d3f1f57461e6360"}
Nov 03 00:18:01.114104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:01.114084    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"
Nov 03 00:18:01.114334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:01.114122    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d369beb32a02e88115dbd5164f70e616f54a256fd8ebdb1d3f1f57461e6360"
Nov 03 00:18:01.227204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:01.227166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:02.813226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.813182    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"]
Nov 03 00:18:02.813802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.813780    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dc3940b3-bdf1-4ca2-9955-fd60a8d74340" containerName="revoke-keys"
Nov 03 00:18:02.813886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.813805    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3940b3-bdf1-4ca2-9955-fd60a8d74340" containerName="revoke-keys"
Nov 03 00:18:02.813964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.813951    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3940b3-bdf1-4ca2-9955-fd60a8d74340" containerName="revoke-keys"
Nov 03 00:18:02.817143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.817121    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.832683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.832660    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8e90c5-openshift-default-dockercfg-6mbt4\""
Nov 03 00:18:02.848857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.848824    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"]
Nov 03 00:18:02.922176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922141    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922198    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfn8\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922313    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922349    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922461    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922485    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922520    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922589    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.922683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.922663    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:02.937752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:02.937725    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:03.023898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.023865    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.023898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.023906    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.023933    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfn8\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.023961    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.023985    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024037    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024069    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024119    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024145    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024343    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024527    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024742    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024790    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.024963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.024943    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.026190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.026168    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.026628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.026608    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.044012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.043987    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.044453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.044410    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfn8\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.129483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.129381    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:03.436917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.436889    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"]
Nov 03 00:18:03.437206 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:18:03.437178    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661be69b_5f30_4354_ab07_3cfefb26c64d.slice/crio-39eb9b54d88c2ce7d2ee02e8cb3c8630fa807e5193e1d60db6e6ac7b37a9390f WatchSource:0}: Error finding container 39eb9b54d88c2ce7d2ee02e8cb3c8630fa807e5193e1d60db6e6ac7b37a9390f: Status 404 returned error can't find the container with id 39eb9b54d88c2ce7d2ee02e8cb3c8630fa807e5193e1d60db6e6ac7b37a9390f
Nov 03 00:18:03.439825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.439788    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:03.439888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.439871    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:03.439930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:03.439914    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:04.130267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:04.130226    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" event={"ID":"661be69b-5f30-4354-ab07-3cfefb26c64d","Type":"ContainerStarted","Data":"01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8"}
Nov 03 00:18:04.130736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:04.130281    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" event={"ID":"661be69b-5f30-4354-ab07-3cfefb26c64d","Type":"ContainerStarted","Data":"39eb9b54d88c2ce7d2ee02e8cb3c8630fa807e5193e1d60db6e6ac7b37a9390f"}
Nov 03 00:18:04.623309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:04.623273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:05.130027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:05.129987    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:05.131967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:05.131926    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 00:18:05.132405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:05.131988    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 00:18:06.130692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:06.130649    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 00:18:06.130879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:06.130714    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 00:18:06.321613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:06.321579    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:07.130729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:07.130690    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 00:18:07.130925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:07.130747    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 00:18:08.123337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:08.123308    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:08.129689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:08.129662    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 00:18:08.129851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:08.129710    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 00:18:08.783073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:08.782985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:09.130640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:09.130550    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Nov 03 00:18:09.130640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:09.130612    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Nov 03 00:18:09.744487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:09.744450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:10.134267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:10.134233    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:10.134797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:10.134776    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:10.135386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:10.135361    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:18:10.206037 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:10.205985    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podStartSLOduration=8.205971558 podStartE2EDuration="8.205971558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:04.185614537 +0000 UTC m=+766.507151340" watchObservedRunningTime="2026-11-03 00:18:10.205971558 +0000 UTC m=+772.527508357"
Nov 03 00:18:11.337316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.337281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:11.730883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.730844    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"]
Nov 03 00:18:11.734515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.734497    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.763544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.763516    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:18:11.763710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.763678    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:18:11.766482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.766457    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"]
Nov 03 00:18:11.834744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.834712    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.834926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.834767    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6n5m\" (UniqueName: \"kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.834926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.834874    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.935634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.935604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.935634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.935650    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6n5m\" (UniqueName: \"kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.935869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.935689    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.936289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.936267    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.937997 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.937974    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:11.955912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:11.955886    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6n5m\" (UniqueName: \"kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m\") pod \"maas-api-revoke-sub-24d5bccf-cfvnb\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:12.047390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:12.047302    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:12.341926 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:18:12.341888    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541ffad3_bf52_4b93_ba74_0913eb5be389.slice/crio-8c43e977470b8b95dfd4c96575c6d0e475f0936b08259134123d9dcb7d0c0b7a WatchSource:0}: Error finding container 8c43e977470b8b95dfd4c96575c6d0e475f0936b08259134123d9dcb7d0c0b7a: Status 404 returned error can't find the container with id 8c43e977470b8b95dfd4c96575c6d0e475f0936b08259134123d9dcb7d0c0b7a
Nov 03 00:18:12.342349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:12.342002    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"]
Nov 03 00:18:13.021090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:13.021003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:13.199445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:13.199398    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb" event={"ID":"541ffad3-bf52-4b93-ba74-0913eb5be389","Type":"ContainerStarted","Data":"53907806108c33b98fe1622b027fccc80301de566fdc06181d9fa0d3778ed834"}
Nov 03 00:18:13.199552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:13.199529    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb" event={"ID":"541ffad3-bf52-4b93-ba74-0913eb5be389","Type":"ContainerStarted","Data":"8c43e977470b8b95dfd4c96575c6d0e475f0936b08259134123d9dcb7d0c0b7a"}
Nov 03 00:18:14.205425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.205387    4550 generic.go:356] "Generic (PLEG): container finished" podID="541ffad3-bf52-4b93-ba74-0913eb5be389" containerID="53907806108c33b98fe1622b027fccc80301de566fdc06181d9fa0d3778ed834" exitCode=0
Nov 03 00:18:14.205950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.205478    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb" event={"ID":"541ffad3-bf52-4b93-ba74-0913eb5be389","Type":"ContainerDied","Data":"53907806108c33b98fe1622b027fccc80301de566fdc06181d9fa0d3778ed834"}
Nov 03 00:18:14.352727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.352698    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:14.466516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.466419    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m\" (UniqueName: \"kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m\") pod \"541ffad3-bf52-4b93-ba74-0913eb5be389\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") "
Nov 03 00:18:14.466516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.466501    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca\") pod \"541ffad3-bf52-4b93-ba74-0913eb5be389\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") "
Nov 03 00:18:14.466759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.466652    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca\") pod \"541ffad3-bf52-4b93-ba74-0913eb5be389\" (UID: \"541ffad3-bf52-4b93-ba74-0913eb5be389\") "
Nov 03 00:18:14.467061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.467026    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca" pod "541ffad3-bf52-4b93-ba74-0913eb5be389" (UID: "541ffad3-bf52-4b93-ba74-0913eb5be389"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:14.468623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.468581    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m" pod "541ffad3-bf52-4b93-ba74-0913eb5be389" (UID: "541ffad3-bf52-4b93-ba74-0913eb5be389"). InnerVolumeSpecName "kube-api-access-l6n5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:14.468822 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.468799    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca" pod "541ffad3-bf52-4b93-ba74-0913eb5be389" (UID: "541ffad3-bf52-4b93-ba74-0913eb5be389"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:14.567701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.567658    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:14.567701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.567699    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6n5m\" (UniqueName: \"kubernetes.io/projected/541ffad3-bf52-4b93-ba74-0913eb5be389-kube-api-access-l6n5m\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:14.567965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.567714    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/541ffad3-bf52-4b93-ba74-0913eb5be389-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:14.626009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:14.625972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:15.212036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:15.211994    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb" event={"ID":"541ffad3-bf52-4b93-ba74-0913eb5be389","Type":"ContainerDied","Data":"8c43e977470b8b95dfd4c96575c6d0e475f0936b08259134123d9dcb7d0c0b7a"}
Nov 03 00:18:15.212480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:15.212047    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"
Nov 03 00:18:15.212480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:15.212055    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c43e977470b8b95dfd4c96575c6d0e475f0936b08259134123d9dcb7d0c0b7a"
Nov 03 00:18:16.241621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:16.241584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:17.706964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:17.706933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:17.809075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:17.809043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:18.077008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:18.076917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:19.060195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:19.060157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:19.970952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:19.970914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:21.823394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:21.823356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:23.529062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:23.529023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:24.722507 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:18:24.722469    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:18:25.316159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.316126    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"]
Nov 03 00:18:25.316610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.316551    4550 cpu_manager.go:414] "containerMap: removing container" podUID="541ffad3-bf52-4b93-ba74-0913eb5be389" containerName="revoke-keys"
Nov 03 00:18:25.316610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.316604    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541ffad3-bf52-4b93-ba74-0913eb5be389" containerName="revoke-keys"
Nov 03 00:18:25.316831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.316710    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="541ffad3-bf52-4b93-ba74-0913eb5be389" containerName="revoke-keys"
Nov 03 00:18:25.320388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.320370    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.338370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.338341    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"]
Nov 03 00:18:25.349060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.349023    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:18:25.349246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.349085    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:18:25.387295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.387266    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.387480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.387322    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvtnq\" (UniqueName: \"kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.387480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.387462    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.410150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.410100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:25.488298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.488261    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.488487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.488328    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvtnq\" (UniqueName: \"kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.488487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.488372    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.489151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.489125    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.490755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.490727    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.508595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.508570    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvtnq\" (UniqueName: \"kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq\") pod \"maas-api-revoke-sub-7af7c9c6-wgzkb\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.633981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.633895    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:25.948704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:25.947210    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"]
Nov 03 00:18:25.952017 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:18:25.951971    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0edebcc_2e4f_4263_b4db_fd9624270794.slice/crio-2febfdc5c8978f6e2954fac096278ac33d57988f955aea8e9cfe6e7500e80250 WatchSource:0}: Error finding container 2febfdc5c8978f6e2954fac096278ac33d57988f955aea8e9cfe6e7500e80250: Status 404 returned error can't find the container with id 2febfdc5c8978f6e2954fac096278ac33d57988f955aea8e9cfe6e7500e80250
Nov 03 00:18:26.280558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:26.280525    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb" event={"ID":"e0edebcc-2e4f-4263-b4db-fd9624270794","Type":"ContainerStarted","Data":"2febfdc5c8978f6e2954fac096278ac33d57988f955aea8e9cfe6e7500e80250"}
Nov 03 00:18:27.227829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:27.227789    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:27.322514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:27.322083    4550 generic.go:356] "Generic (PLEG): container finished" podID="e0edebcc-2e4f-4263-b4db-fd9624270794" containerID="5438459c0a8da21c1eb8868174d37f8d7d23ea7053aa5aed1b930e04e1912e76" exitCode=0
Nov 03 00:18:27.322514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:27.322232    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb" event={"ID":"e0edebcc-2e4f-4263-b4db-fd9624270794","Type":"ContainerDied","Data":"5438459c0a8da21c1eb8868174d37f8d7d23ea7053aa5aed1b930e04e1912e76"}
Nov 03 00:18:28.540054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.540028    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:28.618277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.618172    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:28.630388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.630360    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca\") pod \"e0edebcc-2e4f-4263-b4db-fd9624270794\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") "
Nov 03 00:18:28.630635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.630621    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq\" (UniqueName: \"kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq\") pod \"e0edebcc-2e4f-4263-b4db-fd9624270794\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") "
Nov 03 00:18:28.630726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.630703    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca" pod "e0edebcc-2e4f-4263-b4db-fd9624270794" (UID: "e0edebcc-2e4f-4263-b4db-fd9624270794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:18:28.630826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.630815    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca\") pod \"e0edebcc-2e4f-4263-b4db-fd9624270794\" (UID: \"e0edebcc-2e4f-4263-b4db-fd9624270794\") "
Nov 03 00:18:28.631114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.631103    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:28.635189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.633738    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca" pod "e0edebcc-2e4f-4263-b4db-fd9624270794" (UID: "e0edebcc-2e4f-4263-b4db-fd9624270794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:18:28.635189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.633816    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq" pod "e0edebcc-2e4f-4263-b4db-fd9624270794" (UID: "e0edebcc-2e4f-4263-b4db-fd9624270794"). InnerVolumeSpecName "kube-api-access-xvtnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:18:28.731882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.731848    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0edebcc-2e4f-4263-b4db-fd9624270794-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:28.731882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.731879    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvtnq\" (UniqueName: \"kubernetes.io/projected/e0edebcc-2e4f-4263-b4db-fd9624270794-kube-api-access-xvtnq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:18:28.935619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:28.935587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:29.333468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:29.333373    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"
Nov 03 00:18:29.333468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:29.333372    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb" event={"ID":"e0edebcc-2e4f-4263-b4db-fd9624270794","Type":"ContainerDied","Data":"2febfdc5c8978f6e2954fac096278ac33d57988f955aea8e9cfe6e7500e80250"}
Nov 03 00:18:29.333657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:29.333504    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2febfdc5c8978f6e2954fac096278ac33d57988f955aea8e9cfe6e7500e80250"
Nov 03 00:18:30.514352 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:30.514302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:32.114426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:32.114386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:33.838173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:33.838140    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:35.537240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:35.537205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:37.227674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:37.227644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:37.333141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:37.333094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:39.080653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:39.080611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:40.726800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:40.726752    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:42.515614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:42.515573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:44.331348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:44.330892    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:46.116332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:46.116292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:47.460703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.460668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:47.545723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.545681    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:47.734410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.734332    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:47.884504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.884460    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"]
Nov 03 00:18:47.884883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.884868    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e0edebcc-2e4f-4263-b4db-fd9624270794" containerName="revoke-keys"
Nov 03 00:18:47.884950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.884884    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0edebcc-2e4f-4263-b4db-fd9624270794" containerName="revoke-keys"
Nov 03 00:18:47.884986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.884958    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0edebcc-2e4f-4263-b4db-fd9624270794" containerName="revoke-keys"
Nov 03 00:18:47.888527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.888507    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.909411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.909383    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"]
Nov 03 00:18:47.923051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.913268    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.923051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.920391    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-99bdb8-openshift-default-dockercfg-dq2f2\""
Nov 03 00:18:47.923626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.923606    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"]
Nov 03 00:18:47.939383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.939362    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"]
Nov 03 00:18:47.945052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945028    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945062    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945094    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945162    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945189    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945211    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945374    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945409    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945426    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945512    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945554    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945592    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblk4\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945653    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945693    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:47.945767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945743    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sd4x\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945794    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945824    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:47.945883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:47.945856    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.046472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046368    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.046472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046400    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.046472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046419    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.046472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046465    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046487    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046528    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblk4\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046586    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046646    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046673    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sd4x\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046714    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.046773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046737    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046774    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046802    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046832    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046870    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046893    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046923    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046950    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.047145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.046969    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047210    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047238    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047334    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.047635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047458    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.047635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047494    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.047955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047935    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.048016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.047980    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.048120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.048101    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.048199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.048162    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.049746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.049718    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.050057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.050030    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.050057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.050052    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.050181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.050078    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.067169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.067148    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.067829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.067798    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblk4\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.067931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.067828    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-sc5jp\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.067931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.067886    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sd4x\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.201596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.201553    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:48.230144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.230108    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:48.515146 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:18:48.515111    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd9175e_57ee_4fe2_9cfb_1e3b2eb4faac.slice/crio-927ab1cae1a7f55579c927fa5f0f7e9ffb7d73d807b22641d3108dda8507a3dd WatchSource:0}: Error finding container 927ab1cae1a7f55579c927fa5f0f7e9ffb7d73d807b22641d3108dda8507a3dd: Status 404 returned error can't find the container with id 927ab1cae1a7f55579c927fa5f0f7e9ffb7d73d807b22641d3108dda8507a3dd
Nov 03 00:18:48.515690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.515641    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"]
Nov 03 00:18:48.517415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.517380    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.517534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.517486    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.517605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.517531    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.549018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.548993    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"]
Nov 03 00:18:48.554107 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:18:48.554081    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c133b69_7e30_4bac_b992_cbfeed98e428.slice/crio-35c61277f43bda8e069f8bb6fbdf793eb1176dbb6cf3420a56895a2da8e5d689 WatchSource:0}: Error finding container 35c61277f43bda8e069f8bb6fbdf793eb1176dbb6cf3420a56895a2da8e5d689: Status 404 returned error can't find the container with id 35c61277f43bda8e069f8bb6fbdf793eb1176dbb6cf3420a56895a2da8e5d689
Nov 03 00:18:48.555903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.555866    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.556050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.556032    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:48.556093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:48.556082    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:18:49.451963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.451915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:49.528686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.528646    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" event={"ID":"4c133b69-7e30-4bac-b992-cbfeed98e428","Type":"ContainerStarted","Data":"5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208"}
Nov 03 00:18:49.529174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.528712    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" event={"ID":"4c133b69-7e30-4bac-b992-cbfeed98e428","Type":"ContainerStarted","Data":"35c61277f43bda8e069f8bb6fbdf793eb1176dbb6cf3420a56895a2da8e5d689"}
Nov 03 00:18:49.531121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.531044    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" event={"ID":"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac","Type":"ContainerStarted","Data":"afaecac6aa515470f844e3c074b28d1c76598f8494427e2eca69f125620d32e4"}
Nov 03 00:18:49.531276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.531147    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" event={"ID":"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac","Type":"ContainerStarted","Data":"927ab1cae1a7f55579c927fa5f0f7e9ffb7d73d807b22641d3108dda8507a3dd"}
Nov 03 00:18:49.612457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.602819    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podStartSLOduration=2.602799874 podStartE2EDuration="2.602799874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.600972705 +0000 UTC m=+811.922509562" watchObservedRunningTime="2026-11-03 00:18:49.602799874 +0000 UTC m=+811.924336675"
Nov 03 00:18:49.647494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:49.647426    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podStartSLOduration=2.647412284 podStartE2EDuration="2.647412284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.645407286 +0000 UTC m=+811.966944105" watchObservedRunningTime="2026-11-03 00:18:49.647412284 +0000 UTC m=+811.968949082"
Nov 03 00:18:50.206454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:50.206279    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:50.206677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:50.206459    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.206677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:50.206505    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 00:18:50.231501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:50.231467    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:50.232338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:50.232311    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.232444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:50.232411    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:51.204456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:51.203843    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.204456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:51.203907    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 00:18:51.230557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:51.230519    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.230713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:51.230577    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:51.347223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:51.347192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:52.223455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.210780    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.223455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.210842    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 00:18:52.230861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.230823    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.231015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.230882    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:52.615839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.614159    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"]
Nov 03 00:18:52.635634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.635612    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"]
Nov 03 00:18:52.635829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.635741    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.652687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.652621    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:52.676627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.654206    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"distinct-w4-99bdb8-kserve-self-signed-certs\""
Nov 03 00:18:52.780456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.780189    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.780456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.780252    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.780456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.780318    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.780456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.780356    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.780456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.780411    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.780849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.780468    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkvn\" (UniqueName: \"kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.884517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.884105    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkvn\" (UniqueName: \"kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.884517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.884275    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.884517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.884327    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.884517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.884390    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.884517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.884450    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.884517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.884495    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.893453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.885116    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.893453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.885467    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.893453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.886926    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.895129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.895105    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.904763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.904742    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.908311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.908288    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkvn\" (UniqueName: \"kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn\") pod \"distinct-w4-99bdb8-kserve-bfc6bf467-gklmv\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:52.954553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:52.954518    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:53.144932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.144868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:53.211455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.210491    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.211455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.210551    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 00:18:53.230712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.230673    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.231190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.230735    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:53.382808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.381165    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"]
Nov 03 00:18:53.571465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.566893    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" event={"ID":"c70820b0-0954-433d-a823-e8997a45a058","Type":"ContainerStarted","Data":"38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4"}
Nov 03 00:18:53.571465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.566945    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" event={"ID":"c70820b0-0954-433d-a823-e8997a45a058","Type":"ContainerStarted","Data":"28a6fe5cd8e7fa05f3ba41c40dd9e57d269df4529239b148b8675a62f9aca43b"}
Nov 03 00:18:53.620925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.620849    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podStartSLOduration=1.620828286 podStartE2EDuration="1.620828286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:53.615750346 +0000 UTC m=+815.937287158" watchObservedRunningTime="2026-11-03 00:18:53.620828286 +0000 UTC m=+815.942365088"
Nov 03 00:18:53.955338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:53.955312    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:18:54.209540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:54.206777    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.209540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:54.206836    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Nov 03 00:18:54.231385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:54.231352    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.231752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:54.231401    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:54.733902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:54.733868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:54.744385 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:18:54.744348    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:18:54.980298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:54.980263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:55.231220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:55.231178    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:55.231411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:55.231249    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:55.377900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:55.377862    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:55.379183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:55.379142    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:55.379945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:55.379920    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 00:18:56.230867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:56.230821    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-sc5jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Nov 03 00:18:56.231067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:56.230882    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Nov 03 00:18:56.810108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:56.810079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:18:57.372472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:57.369877    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:57.372472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:57.370564    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:57.372472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:57.371417    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:18:58.566844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:18:58.566807    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:00.188116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:00.188078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:01.912196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:01.912161    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:02.955164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:02.955125    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:19:02.971039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:02.970995    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:19:03.641536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:03.641491    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:19:04.032610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:04.032567    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:05.319792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:05.319759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:05.416884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:05.416843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:05.865801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:05.865768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:05.961025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:05.960973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:07.602699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:07.602667    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:08.938060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:08.938031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:09.251966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:09.251868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:10.906679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:10.906647    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:12.825456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:12.825251    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:14.945944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:14.945894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:16.573192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:16.573153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:18.054133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.054093    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"]
Nov 03 00:19:18.057717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.057689    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.086756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.086723    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"]
Nov 03 00:19:18.112514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112479    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.112679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112605    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.112679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.112813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112774    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r7hf\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.112961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112883    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.112961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112931    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.113106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.112992    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.113106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.113026    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.113106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.113050    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.213718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.213671    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.213891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.213753    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.213891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.213782    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.213891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.213880    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r7hf\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.213953    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.213996    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214032    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214076    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214118    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214217    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214669    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.214886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214762    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.215026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.214983    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.215099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.215076    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.216312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.216285    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.216892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.216873    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.234306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.234284    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.234867    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r7hf\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.244417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.244397    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:18.370673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.370585    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:18.688463 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:19:18.686009    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5815f0_9438_448f_b116_35aaefb1c8b0.slice/crio-349220541f5c4288779431b25e38d125680b482972f30f6eebdaeb49fb648083 WatchSource:0}: Error finding container 349220541f5c4288779431b25e38d125680b482972f30f6eebdaeb49fb648083: Status 404 returned error can't find the container with id 349220541f5c4288779431b25e38d125680b482972f30f6eebdaeb49fb648083
Nov 03 00:19:18.688463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.687227    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"]
Nov 03 00:19:18.696139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.696101    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:18.696223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.696185    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:18.696265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.696226    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:19:18.742347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:18.742311    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" event={"ID":"0e5815f0-9438-448f-b116-35aaefb1c8b0","Type":"ContainerStarted","Data":"349220541f5c4288779431b25e38d125680b482972f30f6eebdaeb49fb648083"}
Nov 03 00:19:19.750220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:19.750176    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" event={"ID":"0e5815f0-9438-448f-b116-35aaefb1c8b0","Type":"ContainerStarted","Data":"4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c"}
Nov 03 00:19:19.829556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:19.829498    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podStartSLOduration=1.829483333 podStartE2EDuration="1.829483333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:19.815107228 +0000 UTC m=+842.136644047" watchObservedRunningTime="2026-11-03 00:19:19.829483333 +0000 UTC m=+842.151020132"
Nov 03 00:19:19.843119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:19.843083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:20.388473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:20.388210    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:20.388473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:20.388370    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:19:20.388473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:20.388416    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 00:19:21.372457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.371828    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:19:21.372457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.371895    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 00:19:21.428880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.428844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:21.449183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.449149    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"]
Nov 03 00:19:21.453317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.453287    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.497459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.492675    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-99bdb8\"/\"distinct-2-w4-99bdb8-kserve-self-signed-certs\""
Nov 03 00:19:21.497459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.496553    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"]
Nov 03 00:19:21.526986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.526951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:21.555294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.555256    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.559541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.555384    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzvkn\" (UniqueName: \"kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.559541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.555467    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.559541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.555555    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.559541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.555579    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.559541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.555636    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666085    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzvkn\" (UniqueName: \"kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666155    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666245    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666269    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666316    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666410    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.667462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.666859    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.678455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.677524    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.682454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.679725    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.689787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.689761    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.703166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.703107    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzvkn\" (UniqueName: \"kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.731878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.731847    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs\") pod \"distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") " pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:21.822453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:21.822271    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:22.190315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.189018    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"]
Nov 03 00:19:22.193055 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:19:22.193000    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod458c7292_7eae_491a_a1c1_4a4bbabe4ea1.slice/crio-62afc6724f8e62bdba16a8a9e81916c13b4484f85cc2ddb526c2f3790f82e40a WatchSource:0}: Error finding container 62afc6724f8e62bdba16a8a9e81916c13b4484f85cc2ddb526c2f3790f82e40a: Status 404 returned error can't find the container with id 62afc6724f8e62bdba16a8a9e81916c13b4484f85cc2ddb526c2f3790f82e40a
Nov 03 00:19:22.384463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.382172    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:19:22.384463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.382241    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 00:19:22.783458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.783351    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" event={"ID":"458c7292-7eae-491a-a1c1-4a4bbabe4ea1","Type":"ContainerStarted","Data":"e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274"}
Nov 03 00:19:22.783458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.783407    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" event={"ID":"458c7292-7eae-491a-a1c1-4a4bbabe4ea1","Type":"ContainerStarted","Data":"62afc6724f8e62bdba16a8a9e81916c13b4484f85cc2ddb526c2f3790f82e40a"}
Nov 03 00:19:22.822744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.822701    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:22.825147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:22.825098    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podStartSLOduration=1.825085426 podStartE2EDuration="1.825085426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:22.823541532 +0000 UTC m=+845.145078332" watchObservedRunningTime="2026-11-03 00:19:22.825085426 +0000 UTC m=+845.146622225"
Nov 03 00:19:23.089174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:23.089070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:23.374236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:23.374133    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:19:23.374599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:23.374561    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 00:19:24.377455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:24.374808    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:19:24.377455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:24.374884    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 00:19:24.724198 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:19:24.724154    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:19:24.859588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:24.859558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:25.371856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:25.371821    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:19:25.372036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:25.371885    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 00:19:26.376043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:26.376009    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:26.376565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:26.376389    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:26.377033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:26.377003    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:19:26.422139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:26.422106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:26.749623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:26.749588    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:26.849505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:26.849476    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:28.358114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:28.358070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:30.014115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:30.014034    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:31.706693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:31.706653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:31.823856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:31.823823    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:31.836820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:31.836796    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:31.849318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:31.849294    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:19:33.440454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:33.440390    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:35.343093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:35.343058    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:36.080481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.065500    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"]
Nov 03 00:19:36.107392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.107356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:36.153730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.153698    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"]
Nov 03 00:19:36.154069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.154017    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.165463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.165415    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.165771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.165753    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.165945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.165926    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.166084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.166069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5m9\" (UniqueName: \"kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.166308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.166291    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.166488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.166473    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.180352 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.180323    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"distinct-2-w0-fc9316-kserve-self-signed-certs\""
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269006    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269080    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269130    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269163    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5m9\" (UniqueName: \"kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269269    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269312    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.269812    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.270467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.270066    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.297348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.297320    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.357858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.335098    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.358907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.358874    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.363277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.363247    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5m9\" (UniqueName: \"kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9\") pod \"distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.523679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.523642    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:36.947694 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:19:36.947657    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776a620a_09d1_44e1_8953_d31795378672.slice/crio-faa60909ef339023d6b8db276a58240bb981444914e62cb1d2f1274ca8dd86dd WatchSource:0}: Error finding container faa60909ef339023d6b8db276a58240bb981444914e62cb1d2f1274ca8dd86dd: Status 404 returned error can't find the container with id faa60909ef339023d6b8db276a58240bb981444914e62cb1d2f1274ca8dd86dd
Nov 03 00:19:36.948485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:36.948459    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"]
Nov 03 00:19:37.218756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:37.218664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:37.515397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:37.515303    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:37.613729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:37.613671    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:37.950328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:37.950292    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" event={"ID":"776a620a-09d1-44e1-8953-d31795378672","Type":"ContainerStarted","Data":"7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6"}
Nov 03 00:19:37.950577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:37.950343    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" event={"ID":"776a620a-09d1-44e1-8953-d31795378672","Type":"ContainerStarted","Data":"faa60909ef339023d6b8db276a58240bb981444914e62cb1d2f1274ca8dd86dd"}
Nov 03 00:19:37.991624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:37.991564    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podStartSLOduration=1.9915497279999999 podStartE2EDuration="1.991549728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:37.990722926 +0000 UTC m=+860.312259747" watchObservedRunningTime="2026-11-03 00:19:37.991549728 +0000 UTC m=+860.313086528"
Nov 03 00:19:38.524654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:38.524617    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:38.797947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:38.797853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:39.066768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:39.066684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:40.748396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:40.748361    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:42.328269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:42.328233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:43.922475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:43.922406    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:45.517011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:45.516973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:46.524746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:46.524698    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:46.541460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:46.541420    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:46.669598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:46.669551    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:47.018694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:47.018649    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:19:47.512454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:47.512408    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:49.558703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:49.558611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:51.328727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:51.328693    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:53.024154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:53.024117    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:54.723521 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:19:54.723397    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:19:54.728601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:54.725419    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:56.451498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:56.451459    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:57.261733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.261699    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"]
Nov 03 00:19:57.265613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.265593    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.283738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.283709    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"]
Nov 03 00:19:57.294217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.294179    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:19:57.294384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.294312    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:19:57.308803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.308766    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.308956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.308840    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.308956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.308942    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhmt\" (UniqueName: \"kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.410522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.410487    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.410738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.410536    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.410738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.410583    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhmt\" (UniqueName: \"kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.411369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.411341    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.413091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.413055    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.430351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.430325    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhmt\" (UniqueName: \"kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt\") pod \"maas-api-revoke-sub-b106dd79-fn7hv\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.579620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.579539    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:19:57.876138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:57.876112    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"]
Nov 03 00:19:57.876261 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:19:57.876143    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811dd584_4e82_4882_a8b4_f2536e2e3ef3.slice/crio-c257aa7a93bfea8caa192d0755874dd37c3f881f042b5ea6758ef85a27d8e744 WatchSource:0}: Error finding container c257aa7a93bfea8caa192d0755874dd37c3f881f042b5ea6758ef85a27d8e744: Status 404 returned error can't find the container with id c257aa7a93bfea8caa192d0755874dd37c3f881f042b5ea6758ef85a27d8e744
Nov 03 00:19:58.033252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:58.033207    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:19:58.146645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:58.146546    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv" event={"ID":"811dd584-4e82-4882-a8b4-f2536e2e3ef3","Type":"ContainerStarted","Data":"c257aa7a93bfea8caa192d0755874dd37c3f881f042b5ea6758ef85a27d8e744"}
Nov 03 00:19:59.153830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:59.153794    4550 generic.go:356] "Generic (PLEG): container finished" podID="811dd584-4e82-4882-a8b4-f2536e2e3ef3" containerID="b8381463874f3ffab044b967e96dca2fd0d2c100da6a5a674a3aaf51eeccb675" exitCode=0
Nov 03 00:19:59.154239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:59.153851    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv" event={"ID":"811dd584-4e82-4882-a8b4-f2536e2e3ef3","Type":"ContainerDied","Data":"b8381463874f3ffab044b967e96dca2fd0d2c100da6a5a674a3aaf51eeccb675"}
Nov 03 00:19:59.828771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:19:59.828727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:00.309332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.309305    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:20:00.442520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.442418    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt\" (UniqueName: \"kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt\") pod \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") "
Nov 03 00:20:00.442520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.442485    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca\") pod \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") "
Nov 03 00:20:00.442728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.442528    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca\") pod \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\" (UID: \"811dd584-4e82-4882-a8b4-f2536e2e3ef3\") "
Nov 03 00:20:00.442968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.442941    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca" pod "811dd584-4e82-4882-a8b4-f2536e2e3ef3" (UID: "811dd584-4e82-4882-a8b4-f2536e2e3ef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:00.444616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.444592    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca" pod "811dd584-4e82-4882-a8b4-f2536e2e3ef3" (UID: "811dd584-4e82-4882-a8b4-f2536e2e3ef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:00.444988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.444971    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt" pod "811dd584-4e82-4882-a8b4-f2536e2e3ef3" (UID: "811dd584-4e82-4882-a8b4-f2536e2e3ef3"). InnerVolumeSpecName "kube-api-access-pnhmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:00.543580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.543548    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhmt\" (UniqueName: \"kubernetes.io/projected/811dd584-4e82-4882-a8b4-f2536e2e3ef3-kube-api-access-pnhmt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:00.543580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.543577    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:00.543580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.543588    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811dd584-4e82-4882-a8b4-f2536e2e3ef3-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:00.918813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.918777    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"]
Nov 03 00:20:00.919182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.919169    4550 cpu_manager.go:414] "containerMap: removing container" podUID="811dd584-4e82-4882-a8b4-f2536e2e3ef3" containerName="revoke-keys"
Nov 03 00:20:00.919182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.919183    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811dd584-4e82-4882-a8b4-f2536e2e3ef3" containerName="revoke-keys"
Nov 03 00:20:00.919320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.919263    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="811dd584-4e82-4882-a8b4-f2536e2e3ef3" containerName="revoke-keys"
Nov 03 00:20:00.922527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.922505    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:00.944882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:00.944853    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"]
Nov 03 00:20:01.049886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.049845    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7f6q\" (UniqueName: \"kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.050125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.050108    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.050173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.050157    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.150961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.150917    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.151122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.150983    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.151122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.151017    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7f6q\" (UniqueName: \"kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.151589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.151564    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.153369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.153344    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.167847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.167815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv" event={"ID":"811dd584-4e82-4882-a8b4-f2536e2e3ef3","Type":"ContainerDied","Data":"c257aa7a93bfea8caa192d0755874dd37c3f881f042b5ea6758ef85a27d8e744"}
Nov 03 00:20:01.167847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.167837    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"
Nov 03 00:20:01.168007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.167867    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c257aa7a93bfea8caa192d0755874dd37c3f881f042b5ea6758ef85a27d8e744"
Nov 03 00:20:01.172021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.171967    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7f6q\" (UniqueName: \"kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q\") pod \"maas-api-revoke-sub-53fb37e5-2qb82\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.238226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.238187    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:01.381533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.381494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:01.543016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.542985    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"]
Nov 03 00:20:01.543781 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:01.543757    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030893b9_f5dd_49e3_9320_849985c23f2a.slice/crio-64b5fb346dd9860e5fe3fd95d03d29dc008f1deacf4fe6f998b218a3971b546c WatchSource:0}: Error finding container 64b5fb346dd9860e5fe3fd95d03d29dc008f1deacf4fe6f998b218a3971b546c: Status 404 returned error can't find the container with id 64b5fb346dd9860e5fe3fd95d03d29dc008f1deacf4fe6f998b218a3971b546c
Nov 03 00:20:01.714308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:01.714273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:02.178415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:02.178373    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82" event={"ID":"030893b9-f5dd-49e3-9320-849985c23f2a","Type":"ContainerStarted","Data":"8f2e61483876ffbd6ee55b788d52a0e1f29334d42ad7c510d7550da121f95806"}
Nov 03 00:20:02.178633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:02.178460    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82" event={"ID":"030893b9-f5dd-49e3-9320-849985c23f2a","Type":"ContainerStarted","Data":"64b5fb346dd9860e5fe3fd95d03d29dc008f1deacf4fe6f998b218a3971b546c"}
Nov 03 00:20:02.218199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:02.218152    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82" podStartSLOduration=2.218138631 podStartE2EDuration="2.218138631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:02.218019365 +0000 UTC m=+884.539556167" watchObservedRunningTime="2026-11-03 00:20:02.218138631 +0000 UTC m=+884.539675432"
Nov 03 00:20:03.102302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.102266    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"]
Nov 03 00:20:03.106621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.106600    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.130375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.130337    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"]
Nov 03 00:20:03.173029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.172994    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6qcw\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173258    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173469    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173576    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173636    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173723    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173769    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173865    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.173972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.173955    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.194947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.194915    4550 generic.go:356] "Generic (PLEG): container finished" podID="030893b9-f5dd-49e3-9320-849985c23f2a" containerID="8f2e61483876ffbd6ee55b788d52a0e1f29334d42ad7c510d7550da121f95806" exitCode=0
Nov 03 00:20:03.195081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.195004    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82" event={"ID":"030893b9-f5dd-49e3-9320-849985c23f2a","Type":"ContainerDied","Data":"8f2e61483876ffbd6ee55b788d52a0e1f29334d42ad7c510d7550da121f95806"}
Nov 03 00:20:03.275187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275146    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6qcw\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.275358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275201    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.275358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275273    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.275358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275345    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.275559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275369    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.275619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275771    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275820    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275898    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275940    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.275968    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.276638    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.276682    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.276774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.276716    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.278867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.278842    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.278988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.278966    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.296803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.296776    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6qcw\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.297149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.297124    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.421866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.421835    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:03.499558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.499407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:03.736309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.736278    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"]
Nov 03 00:20:03.736496 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:03.736477    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c67dd83_f1c1_4a6d_b901_aae9800b7abc.slice/crio-f24acac1b7617ba09e5538526cfd0d0f9fd77edd4000fdbf365450c3b8c50352 WatchSource:0}: Error finding container f24acac1b7617ba09e5538526cfd0d0f9fd77edd4000fdbf365450c3b8c50352: Status 404 returned error can't find the container with id f24acac1b7617ba09e5538526cfd0d0f9fd77edd4000fdbf365450c3b8c50352
Nov 03 00:20:03.739007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.738975    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:03.739089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.739039    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:03.739089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:03.739065    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:04.206285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.206242    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" event={"ID":"5c67dd83-f1c1-4a6d-b901-aae9800b7abc","Type":"ContainerStarted","Data":"91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60"}
Nov 03 00:20:04.206792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.206308    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" event={"ID":"5c67dd83-f1c1-4a6d-b901-aae9800b7abc","Type":"ContainerStarted","Data":"f24acac1b7617ba09e5538526cfd0d0f9fd77edd4000fdbf365450c3b8c50352"}
Nov 03 00:20:04.267118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.267059    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podStartSLOduration=1.2670392129999999 podStartE2EDuration="1.267039213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:04.26188382 +0000 UTC m=+886.583420632" watchObservedRunningTime="2026-11-03 00:20:04.267039213 +0000 UTC m=+886.588576015"
Nov 03 00:20:04.395992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.395959    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:04.422336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.422298    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:04.423875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.423850    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:04.424005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.423898    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:04.494947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.493886    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca\") pod \"030893b9-f5dd-49e3-9320-849985c23f2a\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") "
Nov 03 00:20:04.494947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.493966    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q\" (UniqueName: \"kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q\") pod \"030893b9-f5dd-49e3-9320-849985c23f2a\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") "
Nov 03 00:20:04.494947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.494115    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca\") pod \"030893b9-f5dd-49e3-9320-849985c23f2a\" (UID: \"030893b9-f5dd-49e3-9320-849985c23f2a\") "
Nov 03 00:20:04.495634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.495572    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca" pod "030893b9-f5dd-49e3-9320-849985c23f2a" (UID: "030893b9-f5dd-49e3-9320-849985c23f2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:04.500571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.500550    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca" pod "030893b9-f5dd-49e3-9320-849985c23f2a" (UID: "030893b9-f5dd-49e3-9320-849985c23f2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:04.507784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.506568    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q" pod "030893b9-f5dd-49e3-9320-849985c23f2a" (UID: "030893b9-f5dd-49e3-9320-849985c23f2a"). InnerVolumeSpecName "kube-api-access-s7f6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:04.595360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.595327    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:04.595360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.595358    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/030893b9-f5dd-49e3-9320-849985c23f2a-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:04.595360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:04.595367    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7f6q\" (UniqueName: \"kubernetes.io/projected/030893b9-f5dd-49e3-9320-849985c23f2a-kube-api-access-s7f6q\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:05.215137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:05.215103    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"
Nov 03 00:20:05.215137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:05.215129    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82" event={"ID":"030893b9-f5dd-49e3-9320-849985c23f2a","Type":"ContainerDied","Data":"64b5fb346dd9860e5fe3fd95d03d29dc008f1deacf4fe6f998b218a3971b546c"}
Nov 03 00:20:05.215645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:05.215183    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64b5fb346dd9860e5fe3fd95d03d29dc008f1deacf4fe6f998b218a3971b546c"
Nov 03 00:20:05.422624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:05.422578    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:05.422836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:05.422664    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:05.552295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:05.552205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:06.423725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:06.423668    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:06.424244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:06.423737    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:07.423004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:07.422957    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:07.423184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:07.423020    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:07.837054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:07.836969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:08.423148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:08.423107    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:08.423369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:08.423208    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:08.998334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:08.998292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:09.423017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:09.422984    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:09.423197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:09.423043    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:09.790499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:09.790084    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:09.890549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:09.890505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:09.991972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:09.991935    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:10.422982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:10.422945    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 00:20:10.423387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:10.423009    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 00:20:11.335553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.335520    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"]
Nov 03 00:20:11.335970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.335956    4550 cpu_manager.go:414] "containerMap: removing container" podUID="030893b9-f5dd-49e3-9320-849985c23f2a" containerName="revoke-keys"
Nov 03 00:20:11.336031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.335972    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030893b9-f5dd-49e3-9320-849985c23f2a" containerName="revoke-keys"
Nov 03 00:20:11.336031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.336030    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="030893b9-f5dd-49e3-9320-849985c23f2a" containerName="revoke-keys"
Nov 03 00:20:11.345795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.345758    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.357909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.357877    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"]
Nov 03 00:20:11.368016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.367990    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:11.368149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.368051    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:11.389876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.389841    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshsb\" (UniqueName: \"kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.390024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.389899    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.390126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.390101    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.491295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.491248    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshsb\" (UniqueName: \"kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.491780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.491304    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.491780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.491407    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.492080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.492054    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.493983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.493958    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.511451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.511399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshsb\" (UniqueName: \"kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb\") pod \"maas-api-revoke-sub-f73e534c-lkdkd\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.666581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.666537    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:11.912487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.912396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:11.993857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:11.993772    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"]
Nov 03 00:20:12.263803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:12.263703    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd" event={"ID":"e4eb8a5b-fe48-4074-9280-305ee799afea","Type":"ContainerStarted","Data":"82795683c98162a249b4cdac36a9eaa4373e7d7d7034ca88a36da8c835fd2679"}
Nov 03 00:20:12.464982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:12.464594    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:12.464982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:12.464671    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:12.607235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:12.607150    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:12.607746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:12.607564    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:12.608362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:12.608343    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:20:13.271549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:13.271459    4550 generic.go:356] "Generic (PLEG): container finished" podID="e4eb8a5b-fe48-4074-9280-305ee799afea" containerID="fc0327a3b9891816e2f6957e5cc7882c482102a14676d4807f6031e74178eb99" exitCode=0
Nov 03 00:20:13.271776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:13.271550    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd" event={"ID":"e4eb8a5b-fe48-4074-9280-305ee799afea","Type":"ContainerDied","Data":"fc0327a3b9891816e2f6957e5cc7882c482102a14676d4807f6031e74178eb99"}
Nov 03 00:20:13.730423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:13.730392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:14.442602 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.442572    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:14.528633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.528535    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb\" (UniqueName: \"kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb\") pod \"e4eb8a5b-fe48-4074-9280-305ee799afea\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") "
Nov 03 00:20:14.528803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.528683    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca\") pod \"e4eb8a5b-fe48-4074-9280-305ee799afea\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") "
Nov 03 00:20:14.528873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.528836    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca\") pod \"e4eb8a5b-fe48-4074-9280-305ee799afea\" (UID: \"e4eb8a5b-fe48-4074-9280-305ee799afea\") "
Nov 03 00:20:14.529117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.529073    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca" pod "e4eb8a5b-fe48-4074-9280-305ee799afea" (UID: "e4eb8a5b-fe48-4074-9280-305ee799afea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:14.531087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.531052    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb" pod "e4eb8a5b-fe48-4074-9280-305ee799afea" (UID: "e4eb8a5b-fe48-4074-9280-305ee799afea"). InnerVolumeSpecName "kube-api-access-qshsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:14.531335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.531310    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca" pod "e4eb8a5b-fe48-4074-9280-305ee799afea" (UID: "e4eb8a5b-fe48-4074-9280-305ee799afea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:14.532620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.532595    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qshsb\" (UniqueName: \"kubernetes.io/projected/e4eb8a5b-fe48-4074-9280-305ee799afea-kube-api-access-qshsb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:14.532707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.532624    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:14.532707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:14.532640    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eb8a5b-fe48-4074-9280-305ee799afea-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:15.285335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:15.285301    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"
Nov 03 00:20:15.285335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:15.285318    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd" event={"ID":"e4eb8a5b-fe48-4074-9280-305ee799afea","Type":"ContainerDied","Data":"82795683c98162a249b4cdac36a9eaa4373e7d7d7034ca88a36da8c835fd2679"}
Nov 03 00:20:15.285780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:15.285370    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82795683c98162a249b4cdac36a9eaa4373e7d7d7034ca88a36da8c835fd2679"
Nov 03 00:20:15.399737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:15.399702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:17.057588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:17.057546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:18.203889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.203851    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"]
Nov 03 00:20:18.204833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.204804    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e4eb8a5b-fe48-4074-9280-305ee799afea" containerName="revoke-keys"
Nov 03 00:20:18.204833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.204826    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4eb8a5b-fe48-4074-9280-305ee799afea" containerName="revoke-keys"
Nov 03 00:20:18.205051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.204898    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4eb8a5b-fe48-4074-9280-305ee799afea" containerName="revoke-keys"
Nov 03 00:20:18.208417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.208394    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.231189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.231159    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"]
Nov 03 00:20:18.242877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.242848    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fc9316-openshift-default-dockercfg-8zk4q\""
Nov 03 00:20:18.376439 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376380    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.376593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376575    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5x5\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.376631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376610    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.376691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376640    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.376737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376697    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.376868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376845    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.376935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.376905    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.377036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.377014    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.377087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.377072    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.478697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.478607    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.478697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.478671    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.478919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.478760    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.478919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.478834    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5x5\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.478919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.478869    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.478919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.478896    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479070    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479151    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479173    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479539    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479567    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479658    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.479783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.479735    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.481549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.481523    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.481680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.481660    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.498917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.498896    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.499079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.499059    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5x5\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.526823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.526799    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:18.777728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.777630    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:18.849797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.849467    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"]
Nov 03 00:20:18.853152 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:18.853120    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf147a48b_fba0_4781_a730_9af999130396.slice/crio-47431d7a3557e32ca3efdbe135ecfd1f21326e140df3a8ed4967ba190aaef58f WatchSource:0}: Error finding container 47431d7a3557e32ca3efdbe135ecfd1f21326e140df3a8ed4967ba190aaef58f: Status 404 returned error can't find the container with id 47431d7a3557e32ca3efdbe135ecfd1f21326e140df3a8ed4967ba190aaef58f
Nov 03 00:20:18.856043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.856000    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:18.856165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.856096    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:18.856165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:18.856142    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:19.345451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:19.345052    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" event={"ID":"f147a48b-fba0-4781-a730-9af999130396","Type":"ContainerStarted","Data":"0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f"}
Nov 03 00:20:19.345451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:19.345120    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" event={"ID":"f147a48b-fba0-4781-a730-9af999130396","Type":"ContainerStarted","Data":"47431d7a3557e32ca3efdbe135ecfd1f21326e140df3a8ed4967ba190aaef58f"}
Nov 03 00:20:19.423456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:19.416878    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podStartSLOduration=1.416857727 podStartE2EDuration="1.416857727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:19.415768602 +0000 UTC m=+901.737305427" watchObservedRunningTime="2026-11-03 00:20:19.416857727 +0000 UTC m=+901.738394528"
Nov 03 00:20:19.530459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:19.530366    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:19.530631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:19.530525    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:19.530631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:19.530571    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:20.530469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:20.527825    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:20.530469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:20.527893    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:20.636591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:20.636534    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:20.671742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:20.671699    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:20.826709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:20.826591    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:20.968456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:20.967633    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.006649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:21.006587    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.537950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:21.537904    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:21.539209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:21.538623    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:21.591579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:21.590514    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.015452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.014556    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.015452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.015339    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:20:22.257942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.254582    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:22.257942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.254669    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.531454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.530826    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:22.531454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.530888    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:22.706679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.646180    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.706679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.647257    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:20:22.706679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.648195    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:20:22.706679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.684781    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.761008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.707660    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.871547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.843571    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.871547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.844494    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.930453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:22.907136    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:23.008477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.007313    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:23.029457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.015951    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:23.088689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.088650    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:23.209207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.209169    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:23.337836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.337802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:23.483383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.483282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:23.541773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.541741    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:23.544446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.544398    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:23.552992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.552963    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:20:23.651746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:23.651714    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:20:24.527709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:24.527673    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:24.528118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:24.527749    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:24.737907 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:20:24.737863    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:20:25.179150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:25.179119    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:25.528208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:25.528128    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:25.528208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:25.528181    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:26.234175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.234144    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"]
Nov 03 00:20:26.238259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.238234    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.261815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.257473    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"]
Nov 03 00:20:26.266510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.266482    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:26.266813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.266796    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:26.305451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.305392    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxz5r\" (UniqueName: \"kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.305621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.305557    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.305667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.305619    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.406715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.406671    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.406715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.406721    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.406985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.406815    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxz5r\" (UniqueName: \"kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.407695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.407665    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.409260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.409239    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.427011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.426989    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxz5r\" (UniqueName: \"kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r\") pod \"maas-api-revoke-sub-82ee2558-xnhrt\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.527974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.527877    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:26.527974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.527946    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:26.552826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.552796    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:26.886624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:26.886484    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"]
Nov 03 00:20:26.890404 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:26.890373    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7405e1a3_a1a1_46b8_b673_7eb3f0cde9c9.slice/crio-0835e356aad761d506960fd81fadcb879b03e74fdd38d756733eafefcf5f917a WatchSource:0}: Error finding container 0835e356aad761d506960fd81fadcb879b03e74fdd38d756733eafefcf5f917a: Status 404 returned error can't find the container with id 0835e356aad761d506960fd81fadcb879b03e74fdd38d756733eafefcf5f917a
Nov 03 00:20:27.070451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:27.068005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:27.465946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:27.465847    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt" event={"ID":"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9","Type":"ContainerStarted","Data":"a34ca47b35d5b039a4a8d8af263cd040c7ed718b91b1ba4e84c06fda178ffd19"}
Nov 03 00:20:27.465946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:27.465925    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt" event={"ID":"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9","Type":"ContainerStarted","Data":"0835e356aad761d506960fd81fadcb879b03e74fdd38d756733eafefcf5f917a"}
Nov 03 00:20:27.504954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:27.504867    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt" podStartSLOduration=1.504848551 podStartE2EDuration="1.504848551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:27.504190741 +0000 UTC m=+909.825727541" watchObservedRunningTime="2026-11-03 00:20:27.504848551 +0000 UTC m=+909.826385352"
Nov 03 00:20:27.527735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:27.527692    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Nov 03 00:20:27.527938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:27.527742    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Nov 03 00:20:28.473963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:28.473926    4550 generic.go:356] "Generic (PLEG): container finished" podID="7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" containerID="a34ca47b35d5b039a4a8d8af263cd040c7ed718b91b1ba4e84c06fda178ffd19" exitCode=0
Nov 03 00:20:28.474498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:28.474007    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt" event={"ID":"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9","Type":"ContainerDied","Data":"a34ca47b35d5b039a4a8d8af263cd040c7ed718b91b1ba4e84c06fda178ffd19"}
Nov 03 00:20:28.531416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:28.531390    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:28.531820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:28.531800    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:28.532523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:28.532500    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 00:20:28.747698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:28.747611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:29.633161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.633135    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:29.746888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.746794    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca\") pod \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") "
Nov 03 00:20:29.747067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.746911    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r\" (UniqueName: \"kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r\") pod \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") "
Nov 03 00:20:29.747067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.746959    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca\") pod \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\" (UID: \"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9\") "
Nov 03 00:20:29.747241 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.747215    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca" pod "7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" (UID: "7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:29.747387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.747369    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.749044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.749021    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca" pod "7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" (UID: "7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:29.749129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.749098    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r" pod "7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" (UID: "7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9"). InnerVolumeSpecName "kube-api-access-lxz5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:29.848331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.848289    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxz5r\" (UniqueName: \"kubernetes.io/projected/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-kube-api-access-lxz5r\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:29.848331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:29.848321    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:30.437962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:30.437929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:30.499849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:30.499279    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"
Nov 03 00:20:30.501063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:30.500718    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt" event={"ID":"7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9","Type":"ContainerDied","Data":"0835e356aad761d506960fd81fadcb879b03e74fdd38d756733eafefcf5f917a"}
Nov 03 00:20:30.501063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:30.500775    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0835e356aad761d506960fd81fadcb879b03e74fdd38d756733eafefcf5f917a"
Nov 03 00:20:32.238078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:32.238041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:33.288312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.288274    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"]
Nov 03 00:20:33.309451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.289478    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" containerName="revoke-keys"
Nov 03 00:20:33.336519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.336480    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" containerName="revoke-keys"
Nov 03 00:20:33.353746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.353711    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" containerName="revoke-keys"
Nov 03 00:20:33.385695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.385658    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"]
Nov 03 00:20:33.386101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.386023    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488393    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488470    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488545    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdv2\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488598    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488655    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488684    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488744    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488804    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.509468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.488866    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.608465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.608409    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.617597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.617568    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.618175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.618155    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.626734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.626712    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.627207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.627189    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.628888    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdv2\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.628963    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.626644    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.618048    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.615230    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.630117    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.630902    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.627128    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.638011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.630984    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.704486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.687753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.704486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.688198    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.704486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.688664    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:20:33.705494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.705416    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.723580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.721488    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdv2\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.923762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.923726    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:33.965468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:33.962009    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:34.780351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:34.780310    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"]
Nov 03 00:20:34.831272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:34.829697    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:34.831272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:34.829826    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:34.831272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:34.829871    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:35.538339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:35.538289    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:35.742342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:35.740874    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" event={"ID":"5b469d88-aac1-49ed-9a16-0e375bc165e7","Type":"ContainerStarted","Data":"051cc072ff14609abdf50970af10110034e7041c45a1336bf237049431b194f3"}
Nov 03 00:20:35.742342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:35.740978    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" event={"ID":"5b469d88-aac1-49ed-9a16-0e375bc165e7","Type":"ContainerStarted","Data":"3027a460f7e24fa283cb099a502cce6e47a9e28247bbc45096e0b67c5d0b65ed"}
Nov 03 00:20:35.818735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:35.818633    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podStartSLOduration=2.818617804 podStartE2EDuration="2.818617804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:35.804468774 +0000 UTC m=+918.126005575" watchObservedRunningTime="2026-11-03 00:20:35.818617804 +0000 UTC m=+918.140154604"
Nov 03 00:20:35.924821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:35.924787    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:36.927276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:36.927241    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:20:36.927744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:36.927304    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:20:37.429310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:37.429269    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:37.925684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:37.925649    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:20:37.925853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:37.925700    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:20:38.892780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:38.892687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:38.925404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:38.925370    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:20:38.925609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:38.925423    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:20:39.439972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:39.439934    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:39.924844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:39.924805    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:20:39.925242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:39.924868    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:20:40.925776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:40.925739    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:20:40.926278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:40.925799    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:20:41.135280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.135238    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"]
Nov 03 00:20:41.139596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.139573    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.154788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.154764    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:41.163472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.160130    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvfs\" (UniqueName: \"kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.163472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.160232    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.163472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.160326    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.174012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.173981    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"]
Nov 03 00:20:41.174467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.174446    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:41.261174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.261075    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.261551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.261525    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvfs\" (UniqueName: \"kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.261754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.261735    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.262757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.262731    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.288154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.288125    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvfs\" (UniqueName: \"kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.288701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.288680    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de5e6880-rl4zg\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.313215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.313182    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:41.454970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.454929    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:41.783125 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:41.783092    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a94670_7515_43f1_8218_897123cc4f03.slice/crio-328424c2a531aa01f08cb1dfb8010fc71993f413b9eb8854050e84b3a59292e4 WatchSource:0}: Error finding container 328424c2a531aa01f08cb1dfb8010fc71993f413b9eb8854050e84b3a59292e4: Status 404 returned error can't find the container with id 328424c2a531aa01f08cb1dfb8010fc71993f413b9eb8854050e84b3a59292e4
Nov 03 00:20:41.784369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.784345    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"]
Nov 03 00:20:41.798062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.798028    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg" event={"ID":"61a94670-7515-43f1-8218-897123cc4f03","Type":"ContainerStarted","Data":"328424c2a531aa01f08cb1dfb8010fc71993f413b9eb8854050e84b3a59292e4"}
Nov 03 00:20:41.925316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.925273    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:20:41.925512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:41.925341    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 03 00:20:42.813238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:42.813202    4550 generic.go:356] "Generic (PLEG): container finished" podID="61a94670-7515-43f1-8218-897123cc4f03" containerID="477c81e81df086be3a57de9dc2389a5c5fa5c0d29c1947dd8d0f3e7d27a50b60" exitCode=0
Nov 03 00:20:42.813710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:42.813268    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg" event={"ID":"61a94670-7515-43f1-8218-897123cc4f03","Type":"ContainerDied","Data":"477c81e81df086be3a57de9dc2389a5c5fa5c0d29c1947dd8d0f3e7d27a50b60"}
Nov 03 00:20:42.936310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:42.936265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:43.076395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:43.076312    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:43.076815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:43.076786    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:43.077889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:43.077863    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:20:43.965955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:43.965922    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:44.126457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.126352    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs\" (UniqueName: \"kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs\") pod \"61a94670-7515-43f1-8218-897123cc4f03\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") "
Nov 03 00:20:44.126457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.126447    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca\") pod \"61a94670-7515-43f1-8218-897123cc4f03\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") "
Nov 03 00:20:44.126678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.126503    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca\") pod \"61a94670-7515-43f1-8218-897123cc4f03\" (UID: \"61a94670-7515-43f1-8218-897123cc4f03\") "
Nov 03 00:20:44.126814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.126789    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca" pod "61a94670-7515-43f1-8218-897123cc4f03" (UID: "61a94670-7515-43f1-8218-897123cc4f03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:44.126910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.126894    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61a94670-7515-43f1-8218-897123cc4f03-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:44.128661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.128628    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca" pod "61a94670-7515-43f1-8218-897123cc4f03" (UID: "61a94670-7515-43f1-8218-897123cc4f03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:44.128783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.128759    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs" pod "61a94670-7515-43f1-8218-897123cc4f03" (UID: "61a94670-7515-43f1-8218-897123cc4f03"). InnerVolumeSpecName "kube-api-access-xrvfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:44.228149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.228107    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvfs\" (UniqueName: \"kubernetes.io/projected/61a94670-7515-43f1-8218-897123cc4f03-kube-api-access-xrvfs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:44.228149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.228141    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61a94670-7515-43f1-8218-897123cc4f03-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:44.625684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.625645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:44.737451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.737407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:44.830990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.830959    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"
Nov 03 00:20:44.834522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.834492    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg" event={"ID":"61a94670-7515-43f1-8218-897123cc4f03","Type":"ContainerDied","Data":"328424c2a531aa01f08cb1dfb8010fc71993f413b9eb8854050e84b3a59292e4"}
Nov 03 00:20:44.834714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:44.834697    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="328424c2a531aa01f08cb1dfb8010fc71993f413b9eb8854050e84b3a59292e4"
Nov 03 00:20:46.332559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:46.332521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:48.248467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.248225    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"]
Nov 03 00:20:48.252452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.248981    4550 cpu_manager.go:414] "containerMap: removing container" podUID="61a94670-7515-43f1-8218-897123cc4f03" containerName="revoke-keys"
Nov 03 00:20:48.252452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.249006    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a94670-7515-43f1-8218-897123cc4f03" containerName="revoke-keys"
Nov 03 00:20:48.252452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.249131    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a94670-7515-43f1-8218-897123cc4f03" containerName="revoke-keys"
Nov 03 00:20:48.623527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.622795    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718074    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718137    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718230    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718301    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718336    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718375    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718408    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7gz\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.718454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718453    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.719008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.718492    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.745455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.622664    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"]
Nov 03 00:20:48.745455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.744773    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"]
Nov 03 00:20:48.772457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.770478    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:48.819795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.819758    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.820455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.820414    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.821331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.821312    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.825853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.825833    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.826012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.825995    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7gz\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.826119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.826107    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.826260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.826228    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.826388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.826375    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.826527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.826514    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.826962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.826943    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.827272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.820326    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.829112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.821244    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.829499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.829480    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.829860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.829843    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.866735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.866659    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.874373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.874302    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"]
Nov 03 00:20:48.874740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.874644    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:48.913367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.911971    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.914815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.914771    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7gz\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:48.928145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:48.928110    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028200    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028287    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028319    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028389    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028417    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028456    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028488    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rk9w\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028518    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.033454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.028549    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.118455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.117646    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131053    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131113    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131200    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131234    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131263    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131301    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rk9w\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131339    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131371    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131493    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.131950    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.132631    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.133172    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.133957    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.134655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.134210    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.216875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.215794    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.216875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.216208    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.216875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.216797    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rk9w\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.217729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.217680    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.319945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.319665    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:49.583469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.575334    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:20:49.805164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.725586    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:49.826611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.825682    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:49.874113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.856526    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:49.898766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:49.897882    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:50.049724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.025976    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:50.049724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.037523    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:50.049724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.039612    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Nov 03 00:20:50.199711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.191357    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.205000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.204970    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"]
Nov 03 00:20:50.242658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.242604    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.243022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.242990    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.705748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.705721    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"]
Nov 03 00:20:50.708593 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:50.708548    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d3d58b_f19d_4357_ba0d_f13cfca3bfd4.slice/crio-a190856c71dadbfb11174892e34b18f7249e079b0de6bf1a4441afa75b48fd51 WatchSource:0}: Error finding container a190856c71dadbfb11174892e34b18f7249e079b0de6bf1a4441afa75b48fd51: Status 404 returned error can't find the container with id a190856c71dadbfb11174892e34b18f7249e079b0de6bf1a4441afa75b48fd51
Nov 03 00:20:50.710660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.710617    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.710750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.710711    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.710820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.710766    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:20:50.988516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.987553    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" event={"ID":"b608fd30-9154-443f-9d44-33dc1be9f0e5","Type":"ContainerStarted","Data":"6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046"}
Nov 03 00:20:50.988516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.987626    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" event={"ID":"b608fd30-9154-443f-9d44-33dc1be9f0e5","Type":"ContainerStarted","Data":"69c335bfaf5980168adf591e7d636972aee8df7f456afadacc7e550e18d33afa"}
Nov 03 00:20:50.992455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.990121    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" event={"ID":"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4","Type":"ContainerStarted","Data":"26c205cf9cbe435ab161f9ad6c4d3932697395b02dd6e1fd8c51f1105a2843fc"}
Nov 03 00:20:50.992455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:50.990177    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" event={"ID":"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4","Type":"ContainerStarted","Data":"a190856c71dadbfb11174892e34b18f7249e079b0de6bf1a4441afa75b48fd51"}
Nov 03 00:20:51.033926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.033875    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podStartSLOduration=3.033862167 podStartE2EDuration="3.033862167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:51.030395224 +0000 UTC m=+933.351932041" watchObservedRunningTime="2026-11-03 00:20:51.033862167 +0000 UTC m=+933.355398965"
Nov 03 00:20:51.076457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.073847    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podStartSLOduration=3.073828084 podStartE2EDuration="3.073828084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:51.072929164 +0000 UTC m=+933.394465982" watchObservedRunningTime="2026-11-03 00:20:51.073828084 +0000 UTC m=+933.395364884"
Nov 03 00:20:51.119260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.119227    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:51.121026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.120996    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:51.121178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.121043    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:51.226075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.226040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:51.325535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.322506    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:51.325535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.324862    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:51.325535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.324909    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:51.503024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.502493    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"]
Nov 03 00:20:51.507453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.507355    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.522675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.521984    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"]
Nov 03 00:20:51.523652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.523607    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:20:51.523891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.523870    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:20:51.643630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.641985    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqq5r\" (UniqueName: \"kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.643630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.642049    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.643630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.642174    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.743750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.743075    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.743750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.743173    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqq5r\" (UniqueName: \"kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.743750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.743225    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.743750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.743729    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.745692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.745652    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.783464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.779468    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqq5r\" (UniqueName: \"kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r\") pod \"maas-api-revoke-sub-dd182881-pjkvh\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:51.836870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:51.835620    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:52.122589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:52.122545    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:52.122828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:52.122805    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:52.217096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:52.217004    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"]
Nov 03 00:20:52.227466 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:20:52.226958    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba679af_a218_4a8f_b1d6_965f96f93b4a.slice/crio-99597300bb36d18dc5a57e9c8bf5764125bbf8a222e63df12de494623d9f8693 WatchSource:0}: Error finding container 99597300bb36d18dc5a57e9c8bf5764125bbf8a222e63df12de494623d9f8693: Status 404 returned error can't find the container with id 99597300bb36d18dc5a57e9c8bf5764125bbf8a222e63df12de494623d9f8693
Nov 03 00:20:52.323484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:52.321756    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:52.323484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:52.321818    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:53.043187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.043158    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:53.046444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.046403    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh" event={"ID":"dba679af-a218-4a8f-b1d6-965f96f93b4a","Type":"ContainerStarted","Data":"7e32e2d424c2ea84c4c924565a2266471b48639f0a0d9289512db056c1c49220"}
Nov 03 00:20:53.046577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.046464    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh" event={"ID":"dba679af-a218-4a8f-b1d6-965f96f93b4a","Type":"ContainerStarted","Data":"99597300bb36d18dc5a57e9c8bf5764125bbf8a222e63df12de494623d9f8693"}
Nov 03 00:20:53.112457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.104515    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh" podStartSLOduration=2.104495351 podStartE2EDuration="2.104495351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:53.10188467 +0000 UTC m=+935.423421469" watchObservedRunningTime="2026-11-03 00:20:53.104495351 +0000 UTC m=+935.426032150"
Nov 03 00:20:53.125893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.119055    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:53.125893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.119129    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:53.226458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.216527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:53.320960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.320927    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:53.321114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:53.320979    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:54.052659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.052622    4550 generic.go:356] "Generic (PLEG): container finished" podID="dba679af-a218-4a8f-b1d6-965f96f93b4a" containerID="7e32e2d424c2ea84c4c924565a2266471b48639f0a0d9289512db056c1c49220" exitCode=0
Nov 03 00:20:54.052659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.052659    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh" event={"ID":"dba679af-a218-4a8f-b1d6-965f96f93b4a","Type":"ContainerDied","Data":"7e32e2d424c2ea84c4c924565a2266471b48639f0a0d9289512db056c1c49220"}
Nov 03 00:20:54.121492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.119944    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:54.121492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.120003    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:54.324302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.324209    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:54.324605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.324573    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:54.742297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:54.742262    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:54.825463 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:20:54.825395    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:20:55.129346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.129310    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:55.130000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.129957    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:55.329222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.328854    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:55.329222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.328920    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:55.606529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.591711    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:55.775167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.775128    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca\") pod \"dba679af-a218-4a8f-b1d6-965f96f93b4a\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") "
Nov 03 00:20:55.775947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.775752    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca" pod "dba679af-a218-4a8f-b1d6-965f96f93b4a" (UID: "dba679af-a218-4a8f-b1d6-965f96f93b4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:20:55.776407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.776389    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r\" (UniqueName: \"kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r\") pod \"dba679af-a218-4a8f-b1d6-965f96f93b4a\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") "
Nov 03 00:20:55.776655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.776639    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca\") pod \"dba679af-a218-4a8f-b1d6-965f96f93b4a\" (UID: \"dba679af-a218-4a8f-b1d6-965f96f93b4a\") "
Nov 03 00:20:55.777553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.777534    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:55.817770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.817708    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r" pod "dba679af-a218-4a8f-b1d6-965f96f93b4a" (UID: "dba679af-a218-4a8f-b1d6-965f96f93b4a"). InnerVolumeSpecName "kube-api-access-lqq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:20:55.842774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.842711    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca" pod "dba679af-a218-4a8f-b1d6-965f96f93b4a" (UID: "dba679af-a218-4a8f-b1d6-965f96f93b4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:20:55.893509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.887595    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqq5r\" (UniqueName: \"kubernetes.io/projected/dba679af-a218-4a8f-b1d6-965f96f93b4a-kube-api-access-lqq5r\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:55.893509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:55.887632    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba679af-a218-4a8f-b1d6-965f96f93b4a-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:20:56.104457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.103517    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh" event={"ID":"dba679af-a218-4a8f-b1d6-965f96f93b4a","Type":"ContainerDied","Data":"99597300bb36d18dc5a57e9c8bf5764125bbf8a222e63df12de494623d9f8693"}
Nov 03 00:20:56.104457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.103584    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99597300bb36d18dc5a57e9c8bf5764125bbf8a222e63df12de494623d9f8693"
Nov 03 00:20:56.104457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.103656    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"
Nov 03 00:20:56.121682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.121524    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:56.121682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.121584    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:56.321740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.321705    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:56.322151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.321757    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:56.544878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:56.544836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:57.119079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:57.119042    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:20:57.119299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:57.119100    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 00:20:57.302103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:57.302064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:57.321081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:57.321049    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Nov 03 00:20:57.321362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:57.321332    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Nov 03 00:20:58.138015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.137982    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:58.138629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.138474    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:58.140537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.140511    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:20:58.311110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.311063    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:20:58.324955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.324926    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:58.325398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.325378    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:20:58.326152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:20:58.326130    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:21:00.164132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:00.164044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:02.566928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:02.566880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:03.380893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.380857    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"]
Nov 03 00:21:03.381291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.381277    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dba679af-a218-4a8f-b1d6-965f96f93b4a" containerName="revoke-keys"
Nov 03 00:21:03.381335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.381292    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba679af-a218-4a8f-b1d6-965f96f93b4a" containerName="revoke-keys"
Nov 03 00:21:03.381393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.381383    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba679af-a218-4a8f-b1d6-965f96f93b4a" containerName="revoke-keys"
Nov 03 00:21:03.385149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.385130    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.422461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.422407    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"]
Nov 03 00:21:03.430571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430539    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.430738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430577    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x9gp\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.430738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430609    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.430861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430795    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.430861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430832    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.430861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430856    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.431023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430887    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.431023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430912    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.431023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.430958    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532143    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532207    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532239    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x9gp\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532282    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532401    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532474    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532510    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532552    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532587    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532664    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532712    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.532925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532746    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.533036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.532965    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.533197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.533174    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.534809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.534789    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.535031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.535015    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.553143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.553118    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x9gp\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.553264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.553123    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:03.709854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:03.709818    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:04.028385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.028353    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"]
Nov 03 00:21:04.028590 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:21:04.028567    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1add410_cf1d_4bde_8548_452e7d7f6af4.slice/crio-0a5550304627fa3168223d1d8ff982fbce4bb41f2996d0c32c17edd509f16ce5 WatchSource:0}: Error finding container 0a5550304627fa3168223d1d8ff982fbce4bb41f2996d0c32c17edd509f16ce5: Status 404 returned error can't find the container with id 0a5550304627fa3168223d1d8ff982fbce4bb41f2996d0c32c17edd509f16ce5
Nov 03 00:21:04.031101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.031070    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.031193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.031142    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.031193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.031171    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:04.228191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.228155    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" event={"ID":"a1add410-cf1d-4bde-8548-452e7d7f6af4","Type":"ContainerStarted","Data":"972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85"}
Nov 03 00:21:04.228347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.228210    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" event={"ID":"a1add410-cf1d-4bde-8548-452e7d7f6af4","Type":"ContainerStarted","Data":"0a5550304627fa3168223d1d8ff982fbce4bb41f2996d0c32c17edd509f16ce5"}
Nov 03 00:21:04.272983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.272933    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podStartSLOduration=1.272918521 podStartE2EDuration="1.272918521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:04.269911571 +0000 UTC m=+946.591448385" watchObservedRunningTime="2026-11-03 00:21:04.272918521 +0000 UTC m=+946.594455320"
Nov 03 00:21:04.464962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.464926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:04.710378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:04.710346    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:05.387099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.387046    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"]
Nov 03 00:21:05.392599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.392573    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.395100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.395054    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"]
Nov 03 00:21:05.410804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.408281    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:21:05.421317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.421295    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:21:05.486457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.466171    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6d7\" (UniqueName: \"kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.486994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.486956    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.487178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.487158    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.588278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.588238    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6d7\" (UniqueName: \"kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.588515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.588405    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.588515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.588468    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.589348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.589295    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.591692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.591674    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.616007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.615978    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6d7\" (UniqueName: \"kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7\") pod \"maas-api-revoke-sub-853dc0c3-p66rl\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.711072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.711038    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:05.711498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.711271    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:21:05.711498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:05.711326    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:21:06.041423 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:21:06.041393    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff04cf16_0d69_4721_9f9d_c4509735df70.slice/crio-e1ccba915d238adf4cb46574107f90a6e1d15072c09388423f136bfaa7a77902 WatchSource:0}: Error finding container e1ccba915d238adf4cb46574107f90a6e1d15072c09388423f136bfaa7a77902: Status 404 returned error can't find the container with id e1ccba915d238adf4cb46574107f90a6e1d15072c09388423f136bfaa7a77902
Nov 03 00:21:06.042373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:06.042280    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"]
Nov 03 00:21:06.244981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:06.244943    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl" event={"ID":"ff04cf16-0d69-4721-9f9d-c4509735df70","Type":"ContainerStarted","Data":"e1ccba915d238adf4cb46574107f90a6e1d15072c09388423f136bfaa7a77902"}
Nov 03 00:21:06.326620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:06.326591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:06.710311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:06.710273    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:21:06.710497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:06.710338    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:21:07.252548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:07.252518    4550 generic.go:356] "Generic (PLEG): container finished" podID="ff04cf16-0d69-4721-9f9d-c4509735df70" containerID="3d77ce0ce1acfe987e80b89e5b7c6fea0ebd441e8f5ac879029d53e8cbef227b" exitCode=0
Nov 03 00:21:07.252922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:07.252593    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl" event={"ID":"ff04cf16-0d69-4721-9f9d-c4509735df70","Type":"ContainerDied","Data":"3d77ce0ce1acfe987e80b89e5b7c6fea0ebd441e8f5ac879029d53e8cbef227b"}
Nov 03 00:21:07.710304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:07.710269    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:21:07.710544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:07.710326    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:21:08.082163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.082066    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:08.496423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.496384    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:08.627504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.627386    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca\") pod \"ff04cf16-0d69-4721-9f9d-c4509735df70\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") "
Nov 03 00:21:08.627504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.627460    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7\" (UniqueName: \"kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7\") pod \"ff04cf16-0d69-4721-9f9d-c4509735df70\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") "
Nov 03 00:21:08.627709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.627614    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca\") pod \"ff04cf16-0d69-4721-9f9d-c4509735df70\" (UID: \"ff04cf16-0d69-4721-9f9d-c4509735df70\") "
Nov 03 00:21:08.628144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.628123    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca" pod "ff04cf16-0d69-4721-9f9d-c4509735df70" (UID: "ff04cf16-0d69-4721-9f9d-c4509735df70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:08.630678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.630650    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7" pod "ff04cf16-0d69-4721-9f9d-c4509735df70" (UID: "ff04cf16-0d69-4721-9f9d-c4509735df70"). InnerVolumeSpecName "kube-api-access-6v6d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:08.630956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.630933    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca" pod "ff04cf16-0d69-4721-9f9d-c4509735df70" (UID: "ff04cf16-0d69-4721-9f9d-c4509735df70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:08.710325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.710296    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:21:08.710576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.710550    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:21:08.729015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.728987    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:08.729015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.729013    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff04cf16-0d69-4721-9f9d-c4509735df70-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:08.729193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:08.729023    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6d7\" (UniqueName: \"kubernetes.io/projected/ff04cf16-0d69-4721-9f9d-c4509735df70-kube-api-access-6v6d7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:09.140235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.140199    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:09.291505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.291471    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl" event={"ID":"ff04cf16-0d69-4721-9f9d-c4509735df70","Type":"ContainerDied","Data":"e1ccba915d238adf4cb46574107f90a6e1d15072c09388423f136bfaa7a77902"}
Nov 03 00:21:09.291692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.291513    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"
Nov 03 00:21:09.291692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.291522    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ccba915d238adf4cb46574107f90a6e1d15072c09388423f136bfaa7a77902"
Nov 03 00:21:09.710825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.710793    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:21:09.711314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.711280    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:21:09.792666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:09.792628    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:10.710322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:10.710287    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 00:21:10.710555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:10.710341    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 00:21:11.517333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:11.517294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:11.715693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:11.715660    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:11.716065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:11.716041    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:11.716897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:11.716879    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:21:13.116759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:13.116717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:14.002460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.002412    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"]
Nov 03 00:21:14.003138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.003109    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ff04cf16-0d69-4721-9f9d-c4509735df70" containerName="revoke-keys"
Nov 03 00:21:14.003138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.003133    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff04cf16-0d69-4721-9f9d-c4509735df70" containerName="revoke-keys"
Nov 03 00:21:14.003318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.003284    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff04cf16-0d69-4721-9f9d-c4509735df70" containerName="revoke-keys"
Nov 03 00:21:14.016448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.009754    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.032769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.032743    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"]
Nov 03 00:21:14.041022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.040997    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"distinct-2-w2-4c61c8-kserve-self-signed-certs\""
Nov 03 00:21:14.052089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.052060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:14.099043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.098998    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.099251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.099140    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.099251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.099185    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.099385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.099302    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.099385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.099345    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.099531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.099419    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmgdc\" (UniqueName: \"kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.202224    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.202349    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.202383    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.202487    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.202530    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.202564    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmgdc\" (UniqueName: \"kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.203580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.203124    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.221254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.221222    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.221831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.221739    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.225851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.225597    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.226120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.226068    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.249460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.249320    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmgdc\" (UniqueName: \"kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc\") pod \"distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") " pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.333853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.333754    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:14.632328 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:21:14.632297    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod139b7f6d_22a3_4f8a_ae1f_ebda1a5de641.slice/crio-f787f7172241aaaf0f09bfdbdd8ef1fd9135c3a64ebdcf48712a2d0963f7fbe5 WatchSource:0}: Error finding container f787f7172241aaaf0f09bfdbdd8ef1fd9135c3a64ebdcf48712a2d0963f7fbe5: Status 404 returned error can't find the container with id f787f7172241aaaf0f09bfdbdd8ef1fd9135c3a64ebdcf48712a2d0963f7fbe5
Nov 03 00:21:14.635603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.635574    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"]
Nov 03 00:21:14.826992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:14.826957    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:15.330121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:15.330079    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" event={"ID":"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641","Type":"ContainerStarted","Data":"85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f"}
Nov 03 00:21:15.330121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:15.330126    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" event={"ID":"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641","Type":"ContainerStarted","Data":"f787f7172241aaaf0f09bfdbdd8ef1fd9135c3a64ebdcf48712a2d0963f7fbe5"}
Nov 03 00:21:15.334454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:15.334423    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:15.372391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:15.372340    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podStartSLOduration=2.37232742 podStartE2EDuration="2.37232742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:15.371557145 +0000 UTC m=+957.693093943" watchObservedRunningTime="2026-11-03 00:21:15.37232742 +0000 UTC m=+957.693864217"
Nov 03 00:21:16.055036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.055004    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"]
Nov 03 00:21:16.058791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.058772    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.078748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.078718    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"]
Nov 03 00:21:16.087720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.087692    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:21:16.087864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.087781    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:21:16.132054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.132019    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.132228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.132168    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.132228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.132203    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjnf\" (UniqueName: \"kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.235224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.235172    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.235477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.235348    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.235477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.235402    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjnf\" (UniqueName: \"kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.236376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.236349    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.281453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.268005    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.290461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.289270    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjnf\" (UniqueName: \"kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf\") pod \"maas-api-revoke-sub-f16327f3-v7bsp\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.374278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.373801    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:16.813448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:16.813392    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"]
Nov 03 00:21:17.367713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:17.367514    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp" event={"ID":"472204c2-e688-4adc-83b6-a23e4d5db63c","Type":"ContainerStarted","Data":"97cf87c4ec35928639cbe9c30a0b560434f3e22572f12558d7bf4daf09e93270"}
Nov 03 00:21:17.367713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:17.367614    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp" event={"ID":"472204c2-e688-4adc-83b6-a23e4d5db63c","Type":"ContainerStarted","Data":"45d96fb87e39409daf2835849103e24d8a960b60f4a740a5fe3588179acd7a2a"}
Nov 03 00:21:17.430228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:17.430156    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp" podStartSLOduration=1.430137425 podStartE2EDuration="1.430137425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:17.426053195 +0000 UTC m=+959.747589994" watchObservedRunningTime="2026-11-03 00:21:17.430137425 +0000 UTC m=+959.751674224"
Nov 03 00:21:17.843165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:17.843071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:18.431472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:18.430603    4550 generic.go:356] "Generic (PLEG): container finished" podID="472204c2-e688-4adc-83b6-a23e4d5db63c" containerID="97cf87c4ec35928639cbe9c30a0b560434f3e22572f12558d7bf4daf09e93270" exitCode=0
Nov 03 00:21:18.431472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:18.430853    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp" event={"ID":"472204c2-e688-4adc-83b6-a23e4d5db63c","Type":"ContainerDied","Data":"97cf87c4ec35928639cbe9c30a0b560434f3e22572f12558d7bf4daf09e93270"}
Nov 03 00:21:19.611681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.611650    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:19.705279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.705204    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca\") pod \"472204c2-e688-4adc-83b6-a23e4d5db63c\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") "
Nov 03 00:21:19.705442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.705417    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf\" (UniqueName: \"kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf\") pod \"472204c2-e688-4adc-83b6-a23e4d5db63c\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") "
Nov 03 00:21:19.705501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.705482    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca\") pod \"472204c2-e688-4adc-83b6-a23e4d5db63c\" (UID: \"472204c2-e688-4adc-83b6-a23e4d5db63c\") "
Nov 03 00:21:19.705672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.705643    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca" pod "472204c2-e688-4adc-83b6-a23e4d5db63c" (UID: "472204c2-e688-4adc-83b6-a23e4d5db63c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:19.705978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.705953    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:19.707888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.707863    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca" pod "472204c2-e688-4adc-83b6-a23e4d5db63c" (UID: "472204c2-e688-4adc-83b6-a23e4d5db63c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:19.708024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.707933    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf" pod "472204c2-e688-4adc-83b6-a23e4d5db63c" (UID: "472204c2-e688-4adc-83b6-a23e4d5db63c"). InnerVolumeSpecName "kube-api-access-fgjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:19.807177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.807135    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjnf\" (UniqueName: \"kubernetes.io/projected/472204c2-e688-4adc-83b6-a23e4d5db63c-kube-api-access-fgjnf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:19.807177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.807165    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472204c2-e688-4adc-83b6-a23e4d5db63c-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:19.917989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:19.917959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:20.446002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:20.445970    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"
Nov 03 00:21:20.446152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:20.445962    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp" event={"ID":"472204c2-e688-4adc-83b6-a23e4d5db63c","Type":"ContainerDied","Data":"45d96fb87e39409daf2835849103e24d8a960b60f4a740a5fe3588179acd7a2a"}
Nov 03 00:21:20.446152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:20.446095    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d96fb87e39409daf2835849103e24d8a960b60f4a740a5fe3588179acd7a2a"
Nov 03 00:21:21.528749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:21.528711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:23.192120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:23.192075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:24.334553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:24.334503    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:24.347368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:24.347340    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:24.483226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:24.483196    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:24.724487 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:21:24.724447    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:21:24.914584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:24.914544    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:26.529928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:26.529885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:28.681208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:28.681170    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:28.689716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:28.689684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:28.835469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:28.832747    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:28.859456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:28.858778    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:28.899645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:28.899592    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.019695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.019582    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.124655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.124598    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:29.125007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.124975    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.374697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.374577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:29.484934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.484154    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.564500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.564442    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.578488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.577521    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:29.578488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.577587    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.635653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.635545    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.645640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.645599    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.691650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.691589    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.692701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.692676    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:21:29.885534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.885483    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.886155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.886088    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.916132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.915327    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.934785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.934751    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:21:29.936740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.936711    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:21:29.936953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:29.936939    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:21:30.044560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.042512    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.046675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.046635    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.445455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.430022    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:30.489620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.489551    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.490667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.490640    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:30.584604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.573650    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.584604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.574464    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:21:30.698524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.673873    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.717453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.699906    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:21:30.824643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.819002    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.860313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.841568    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:30.904451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.900565    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.904451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.900957    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.923457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:30.922524    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.171494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.168083    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:21:31.217904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.217874    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:21:31.251138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.251106    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:21:31.496507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.493475    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.699693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.699629    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:21:31.780868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.780789    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:21:31.924461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:31.923630    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 00:21:32.003272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:32.003240    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:21:32.010489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:32.010461    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:21:32.034551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:32.034034    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:21:33.225301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.225266    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:33.401093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.401061    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"]
Nov 03 00:21:33.401552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.401539    4550 cpu_manager.go:414] "containerMap: removing container" podUID="472204c2-e688-4adc-83b6-a23e4d5db63c" containerName="revoke-keys"
Nov 03 00:21:33.401609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.401554    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472204c2-e688-4adc-83b6-a23e4d5db63c" containerName="revoke-keys"
Nov 03 00:21:33.401650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.401639    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="472204c2-e688-4adc-83b6-a23e4d5db63c" containerName="revoke-keys"
Nov 03 00:21:33.406335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.406316    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.424677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.424645    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4c61c8-openshift-default-dockercfg-sclgr\""
Nov 03 00:21:33.441117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.441087    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"]
Nov 03 00:21:33.441589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.441560    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.441744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.441724    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stpxs\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.441870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.441849    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.441931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.441911    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.441985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.441938    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.442108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.442085    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.442194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.442175    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.442296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.442211    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.442296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.442227    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543018    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543112    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stpxs\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543171    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543251    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543290    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543346    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543372    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543394    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.543920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.543881    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.544255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.544224    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.546071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.545221    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.546071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.546010    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.546887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.546866    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.547557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.547537    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.547986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.547965    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.565455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.565323    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stpxs\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.567122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.567099    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:33.721042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:33.721010    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:34.045393 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:21:34.045359    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12249439_f90c_4cb0_a78b_592da71491be.slice/crio-d32684d1b84f53d7727e08ffec00961d6ff372ea05d8251aa0028cd7799b1d05 WatchSource:0}: Error finding container d32684d1b84f53d7727e08ffec00961d6ff372ea05d8251aa0028cd7799b1d05: Status 404 returned error can't find the container with id d32684d1b84f53d7727e08ffec00961d6ff372ea05d8251aa0028cd7799b1d05
Nov 03 00:21:34.045541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.045488    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"]
Nov 03 00:21:34.048374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.048339    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:34.048488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.048417    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:34.048535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.048500    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:34.816516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.816475    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" event={"ID":"12249439-f90c-4cb0-a78b-592da71491be","Type":"ContainerStarted","Data":"7ce606b7131cf57a5025ba4fdb52858411c7163c59e14ae0fcffb91f2318ad60"}
Nov 03 00:21:34.816936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.816539    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" event={"ID":"12249439-f90c-4cb0-a78b-592da71491be","Type":"ContainerStarted","Data":"d32684d1b84f53d7727e08ffec00961d6ff372ea05d8251aa0028cd7799b1d05"}
Nov 03 00:21:34.878454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:34.877595    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podStartSLOduration=1.877573462 podStartE2EDuration="1.877573462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:34.874924496 +0000 UTC m=+977.196461326" watchObservedRunningTime="2026-11-03 00:21:34.877573462 +0000 UTC m=+977.199110266"
Nov 03 00:21:35.004423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:35.004376    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:35.721357 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:35.721322    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:35.722935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:35.722908    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:21:35.723057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:35.722966    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:21:36.489613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:36.487891    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:36.590385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:36.590344    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:36.721415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:36.721376    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:21:36.721598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:36.721457    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:21:36.838936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:36.838850    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:37.722531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:37.722491    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:21:37.722927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:37.722557    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:21:37.729365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:37.729340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:37.988146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:37.988027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:38.298960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:38.298879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:38.398544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:38.398510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:38.672460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:38.669097    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:38.721989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:38.721959    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:21:38.722191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:38.722020    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:21:39.722060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:39.722028    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:21:39.722513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:39.722081    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:21:40.603081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:40.603004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:40.721578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:40.721541    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:21:40.721782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:40.721598    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Nov 03 00:21:41.726216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:41.726179    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:41.726667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:41.726476    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:41.727171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:41.727150    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:21:42.226942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:42.226869    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:44.055616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:44.055560    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.193125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:44.193085    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:44.193492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:44.193407    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.551051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:44.550579    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:45.062315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:45.060504    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:45.062315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:45.061530    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:45.063290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:45.063255    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:45.063787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:45.063767    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:21:45.312790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:45.291308    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:21:46.691177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:46.691145    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:48.351547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.351503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:48.484739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.484699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:48.519048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.519008    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"]
Nov 03 00:21:48.524960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.524933    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.541357 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.541329    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.541688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.541664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.541973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.541951    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.542082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.541986    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbg74\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.542082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.542032    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.542082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.542062    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.542553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.542117    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.542553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.542175    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.542553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.542215    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.547114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.547091    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"]
Nov 03 00:21:48.551020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.550996    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.556217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.556190    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b85343-openshift-default-dockercfg-jnjww\""
Nov 03 00:21:48.609532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.609069    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"]
Nov 03 00:21:48.613585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.613555    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"]
Nov 03 00:21:48.642963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.642925    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.642963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.642964    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643009    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643031    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klg67\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643075    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643095    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643118    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643156    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643179    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643213    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643235    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643254    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643299    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643367    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643391    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643408    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbg74\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.643808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643425    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.643808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643480    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.644108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.643827    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.644846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.644816    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.645214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.645188    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.646223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.645419    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.646223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.645754    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.647025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.647000    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.649920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.649897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.673474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.673265    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.676179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.676147    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbg74\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.744416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744379    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.744638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744478    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.744638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744500    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.744638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744543    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.744638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744630    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.744841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744664    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744718    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.745412    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klg67\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744854    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.744870    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.745329    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.745070    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.745524    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.745776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.745750    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.747126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.747107    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.747949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.747927    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.757089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.757048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:48.766870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.766846    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.767186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.767162    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klg67\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.843070 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.843037    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:48.866235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.866158    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:48.919216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:48.918654    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:49.155741 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:21:49.155672    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a5789a_6d7c_43a4_916d_c554555dbb04.slice/crio-88e8acfff6195d81584635de12673ad182d866a6ff442c93cd29e7d4160fac2c WatchSource:0}: Error finding container 88e8acfff6195d81584635de12673ad182d866a6ff442c93cd29e7d4160fac2c: Status 404 returned error can't find the container with id 88e8acfff6195d81584635de12673ad182d866a6ff442c93cd29e7d4160fac2c
Nov 03 00:21:49.157565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.157538    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"]
Nov 03 00:21:49.158696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.158656    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.158804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.158736    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.158804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.158779    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.190836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.190811    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"]
Nov 03 00:21:49.195591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.195555    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.195703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.195642    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.195703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.195682    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:21:49.790295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.790243    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"]
Nov 03 00:21:49.821529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.820661    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"]
Nov 03 00:21:49.822796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.822767    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:49.839570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.838633    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:21:49.839570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.838804    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:21:49.881456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.867486    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:49.881456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.875057    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:49.881456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.875330    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzp5w\" (UniqueName: \"kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:49.982454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.982151    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzp5w\" (UniqueName: \"kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:49.982454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.982316    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:49.982454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.982405    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:50.001480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.983463    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:50.001480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.993617    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" event={"ID":"e8a5789a-6d7c-43a4-916d-c554555dbb04","Type":"ContainerStarted","Data":"8eaec5d73144fe1791a459a908bd2affcce88a4d84fbc607d182d0dd439c6e0d"}
Nov 03 00:21:50.001480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.993692    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" event={"ID":"e8a5789a-6d7c-43a4-916d-c554555dbb04","Type":"ContainerStarted","Data":"88e8acfff6195d81584635de12673ad182d866a6ff442c93cd29e7d4160fac2c"}
Nov 03 00:21:50.001480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.996902    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" event={"ID":"6922f8cf-c8bf-464d-97f3-60bec085cddd","Type":"ContainerStarted","Data":"669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895"}
Nov 03 00:21:50.001480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:49.996959    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" event={"ID":"6922f8cf-c8bf-464d-97f3-60bec085cddd","Type":"ContainerStarted","Data":"67c1783b7c99c355b0935e57edd912d600d4b3e1e0402128854a765a57eb918e"}
Nov 03 00:21:50.007255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.003950    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:50.011010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.009768    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzp5w\" (UniqueName: \"kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w\") pod \"maas-api-revoke-sub-c433e695-b7rrj\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:50.044821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.044723    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podStartSLOduration=2.04470857 podStartE2EDuration="2.04470857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:50.043580583 +0000 UTC m=+992.365117382" watchObservedRunningTime="2026-11-03 00:21:50.04470857 +0000 UTC m=+992.366245411"
Nov 03 00:21:50.140200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.140145    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podStartSLOduration=2.140128121 podStartE2EDuration="2.140128121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:50.13712241 +0000 UTC m=+992.458659240" watchObservedRunningTime="2026-11-03 00:21:50.140128121 +0000 UTC m=+992.461664921"
Nov 03 00:21:50.195871 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.194674    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:50.547910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.547864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:50.613520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.606275    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"]
Nov 03 00:21:50.613520 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:21:50.609797    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4202c346_e09f_4152_bfae_9b609acba24b.slice/crio-2e75192445e5aac07f5c8d73f2155913d2ea07c0253eec432e35a689802b0463 WatchSource:0}: Error finding container 2e75192445e5aac07f5c8d73f2155913d2ea07c0253eec432e35a689802b0463: Status 404 returned error can't find the container with id 2e75192445e5aac07f5c8d73f2155913d2ea07c0253eec432e35a689802b0463
Nov 03 00:21:50.845020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.844168    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:50.845390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.845235    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:21:50.845390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.845275    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 00:21:50.879536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.878206    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:50.879536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.879385    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:21:50.879536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:50.879450    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:21:51.032535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:51.032486    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj" event={"ID":"4202c346-e09f-4152-bfae-9b609acba24b","Type":"ContainerStarted","Data":"2e75192445e5aac07f5c8d73f2155913d2ea07c0253eec432e35a689802b0463"}
Nov 03 00:21:51.843806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:51.843778    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:21:51.843952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:51.843833    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 00:21:51.874656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:51.874533    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:21:51.874656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:51.874598    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:21:52.043080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.043041    4550 generic.go:356] "Generic (PLEG): container finished" podID="4202c346-e09f-4152-bfae-9b609acba24b" containerID="8daeb0b4d7d716b8ef2243b1f0882d9365c6b9eb6368c89d4fceba86098e6456" exitCode=0
Nov 03 00:21:52.043251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.043114    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj" event={"ID":"4202c346-e09f-4152-bfae-9b609acba24b","Type":"ContainerDied","Data":"8daeb0b4d7d716b8ef2243b1f0882d9365c6b9eb6368c89d4fceba86098e6456"}
Nov 03 00:21:52.443343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.443294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:52.844055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.843966    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:21:52.844055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.844035    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 00:21:52.871456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.867852    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:21:52.871456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:52.867938    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:21:53.344509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.344485    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:53.378529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.355419    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca\") pod \"4202c346-e09f-4152-bfae-9b609acba24b\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") "
Nov 03 00:21:53.378729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.378686    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca\") pod \"4202c346-e09f-4152-bfae-9b609acba24b\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") "
Nov 03 00:21:53.378794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.378730    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w\" (UniqueName: \"kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w\") pod \"4202c346-e09f-4152-bfae-9b609acba24b\" (UID: \"4202c346-e09f-4152-bfae-9b609acba24b\") "
Nov 03 00:21:53.378871 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.376666    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca" pod "4202c346-e09f-4152-bfae-9b609acba24b" (UID: "4202c346-e09f-4152-bfae-9b609acba24b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:53.379389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.379357    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:53.422510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.417869    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w" pod "4202c346-e09f-4152-bfae-9b609acba24b" (UID: "4202c346-e09f-4152-bfae-9b609acba24b"). InnerVolumeSpecName "kube-api-access-kzp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:53.427827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.427648    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca" pod "4202c346-e09f-4152-bfae-9b609acba24b" (UID: "4202c346-e09f-4152-bfae-9b609acba24b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:53.484456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.483391    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4202c346-e09f-4152-bfae-9b609acba24b-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:53.508456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.504495    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzp5w\" (UniqueName: \"kubernetes.io/projected/4202c346-e09f-4152-bfae-9b609acba24b-kube-api-access-kzp5w\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:53.843775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.843739    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:21:53.843935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.843800    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 00:21:53.884721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.884678    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"]
Nov 03 00:21:53.901456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.888904    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:21:53.901456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.888980    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:21:53.908526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.908495    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4202c346-e09f-4152-bfae-9b609acba24b" containerName="revoke-keys"
Nov 03 00:21:53.908747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.908730    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4202c346-e09f-4152-bfae-9b609acba24b" containerName="revoke-keys"
Nov 03 00:21:53.909021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.909007    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4202c346-e09f-4152-bfae-9b609acba24b" containerName="revoke-keys"
Nov 03 00:21:53.928001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.927972    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"]
Nov 03 00:21:53.928335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:53.928273    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.024366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.024325    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.024650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.024403    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9fz\" (UniqueName: \"kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.024650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.024468    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.108781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.108698    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj" event={"ID":"4202c346-e09f-4152-bfae-9b609acba24b","Type":"ContainerDied","Data":"2e75192445e5aac07f5c8d73f2155913d2ea07c0253eec432e35a689802b0463"}
Nov 03 00:21:54.108934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.108823    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e75192445e5aac07f5c8d73f2155913d2ea07c0253eec432e35a689802b0463"
Nov 03 00:21:54.108934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.108780    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"
Nov 03 00:21:54.125785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.125755    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.125946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.125813    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9fz\" (UniqueName: \"kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.125946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.125848    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.126596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.126520    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.129265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.129134    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.145936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.145911    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9fz\" (UniqueName: \"kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz\") pod \"maas-api-revoke-sub-c0aee6b3-v4zj9\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.245538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.245504    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:54.250799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.250767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:54.696552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.679590    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"]
Nov 03 00:21:54.751328 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:21:54.751288    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:21:54.843881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.843638    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:21:54.843881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.843684    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 00:21:54.871459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.871357    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:21:54.871459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:54.871422    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:21:55.124929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:55.123961    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9" event={"ID":"8567e234-60b5-4364-a0b7-68b83abd51ec","Type":"ContainerStarted","Data":"c8ea5402792b6302a3695acd090818d7091c7afae8a8387d4ae66016d304a1d1"}
Nov 03 00:21:55.843862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:55.843826    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 00:21:55.844284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:55.843879    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 00:21:55.866610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:55.866574    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:21:55.866784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:55.866637    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 00:21:56.059827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.059791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:56.138236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.138136    4550 generic.go:356] "Generic (PLEG): container finished" podID="8567e234-60b5-4364-a0b7-68b83abd51ec" containerID="eddd7b61aa4b1aebf5e96e5a9025f0ce1953f6158f6899ad08c83a64111205ff" exitCode=0
Nov 03 00:21:56.138396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.138228    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9" event={"ID":"8567e234-60b5-4364-a0b7-68b83abd51ec","Type":"ContainerDied","Data":"eddd7b61aa4b1aebf5e96e5a9025f0ce1953f6158f6899ad08c83a64111205ff"}
Nov 03 00:21:56.849125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.849088    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:56.850035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.850014    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:56.850734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.850710    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 00:21:56.908760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.908731    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:56.909265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.909224    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:56.911226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:56.911203    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:21:57.313436 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.313399    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:57.417800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.417758    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca\") pod \"8567e234-60b5-4364-a0b7-68b83abd51ec\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") "
Nov 03 00:21:57.417982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.417867    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz\" (UniqueName: \"kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz\") pod \"8567e234-60b5-4364-a0b7-68b83abd51ec\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") "
Nov 03 00:21:57.417982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.417895    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca\") pod \"8567e234-60b5-4364-a0b7-68b83abd51ec\" (UID: \"8567e234-60b5-4364-a0b7-68b83abd51ec\") "
Nov 03 00:21:57.418225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.418182    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca" pod "8567e234-60b5-4364-a0b7-68b83abd51ec" (UID: "8567e234-60b5-4364-a0b7-68b83abd51ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:21:57.420000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.419975    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz" pod "8567e234-60b5-4364-a0b7-68b83abd51ec" (UID: "8567e234-60b5-4364-a0b7-68b83abd51ec"). InnerVolumeSpecName "kube-api-access-xv9fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:21:57.420127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.420112    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca" pod "8567e234-60b5-4364-a0b7-68b83abd51ec" (UID: "8567e234-60b5-4364-a0b7-68b83abd51ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:21:57.519172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.519138    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:57.519172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.519168    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv9fz\" (UniqueName: \"kubernetes.io/projected/8567e234-60b5-4364-a0b7-68b83abd51ec-kube-api-access-xv9fz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:57.519172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.519179    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8567e234-60b5-4364-a0b7-68b83abd51ec-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:21:57.796243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:57.796162    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:58.152382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:58.152351    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"
Nov 03 00:21:58.152784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:58.152383    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9" event={"ID":"8567e234-60b5-4364-a0b7-68b83abd51ec","Type":"ContainerDied","Data":"c8ea5402792b6302a3695acd090818d7091c7afae8a8387d4ae66016d304a1d1"}
Nov 03 00:21:58.152784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:58.152469    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ea5402792b6302a3695acd090818d7091c7afae8a8387d4ae66016d304a1d1"
Nov 03 00:21:59.415122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:59.415086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:21:59.656588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:21:59.656546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:01.026107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:01.026071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:01.490389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:01.490342    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:01.590411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:01.590365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:01.719464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:01.713966    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:02.641412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:02.641381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:02.758359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:02.758326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:02.886032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:02.885999    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:03.286481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.286410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:03.573509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.573402    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"]
Nov 03 00:22:03.579733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.579675    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8567e234-60b5-4364-a0b7-68b83abd51ec" containerName="revoke-keys"
Nov 03 00:22:03.579733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.579709    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8567e234-60b5-4364-a0b7-68b83abd51ec" containerName="revoke-keys"
Nov 03 00:22:03.579977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.579945    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8567e234-60b5-4364-a0b7-68b83abd51ec" containerName="revoke-keys"
Nov 03 00:22:03.584553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.584523    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.619489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.619422    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"]
Nov 03 00:22:03.716661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716619    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716695    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716759    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716823    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716852    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716870    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.716981    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.717023    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.717208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.717041    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwt6x\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.817983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.817935    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818030    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818071    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818117    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818158    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818187    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818267    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818327    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818349    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818352    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwt6x\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818413    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818508    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.818887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818865    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.819001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.818976    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.820707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.820677    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.820904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.820889    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.842198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.842115    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.842331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.842239    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwt6x\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:03.922671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:03.922631    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:04.242010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.241980    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"]
Nov 03 00:22:04.242779 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:22:04.242750    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad9f8fd_b0cd_45ed_a8e0_89e42edbe163.slice/crio-508fb434d1ba89555dcffca201259683bc008832d61e34e06c641b76d93a9365 WatchSource:0}: Error finding container 508fb434d1ba89555dcffca201259683bc008832d61e34e06c641b76d93a9365: Status 404 returned error can't find the container with id 508fb434d1ba89555dcffca201259683bc008832d61e34e06c641b76d93a9365
Nov 03 00:22:04.245392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.245356    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:04.245494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.245450    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:04.245494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.245489    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:04.259560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.259535    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"]
Nov 03 00:22:04.265145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.265124    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.280019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.279994    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"]
Nov 03 00:22:04.285790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.285761    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:22:04.286838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.286091    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:22:04.426228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.426184    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.426378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.426298    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.426507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.426488    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkb9\" (UniqueName: \"kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.528212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.528128    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkb9\" (UniqueName: \"kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.528372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.528232    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.528372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.528295    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.528991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.528969    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.530916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.530889    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.547942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.547911    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkb9\" (UniqueName: \"kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9\") pod \"maas-api-revoke-sub-a0397f2d-hrchl\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.611587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.611554    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:04.922519 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:22:04.922485    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e5c23d_389c_44b3_83b3_3264e8d0ffa1.slice/crio-fe5157518198cfea18785eed796af5402dbe238dec5774274218d19eb42dff3d WatchSource:0}: Error finding container fe5157518198cfea18785eed796af5402dbe238dec5774274218d19eb42dff3d: Status 404 returned error can't find the container with id fe5157518198cfea18785eed796af5402dbe238dec5774274218d19eb42dff3d
Nov 03 00:22:04.924324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:04.924302    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"]
Nov 03 00:22:05.247670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.247576    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" event={"ID":"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163","Type":"ContainerStarted","Data":"243527cb1579e1365f23736396330d2f8a9f1372ac0c08dc73e6852638436923"}
Nov 03 00:22:05.247670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.247640    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" event={"ID":"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163","Type":"ContainerStarted","Data":"508fb434d1ba89555dcffca201259683bc008832d61e34e06c641b76d93a9365"}
Nov 03 00:22:05.249004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.248977    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl" event={"ID":"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1","Type":"ContainerStarted","Data":"fe5157518198cfea18785eed796af5402dbe238dec5774274218d19eb42dff3d"}
Nov 03 00:22:05.326634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.326576    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podStartSLOduration=2.326560501 podStartE2EDuration="2.326560501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:05.31347689 +0000 UTC m=+1007.635013697" watchObservedRunningTime="2026-11-03 00:22:05.326560501 +0000 UTC m=+1007.648097301"
Nov 03 00:22:05.541961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.541875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:05.923748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.923705    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:05.925373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.925342    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:22:05.925547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:05.925403    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:22:06.286914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:06.286806    4550 generic.go:356] "Generic (PLEG): container finished" podID="a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" containerID="c0895d8d19a1ea78b946a09c50b8461186ed9ce34020338f8dd2d3c58c284b55" exitCode=0
Nov 03 00:22:06.287077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:06.286898    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl" event={"ID":"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1","Type":"ContainerDied","Data":"c0895d8d19a1ea78b946a09c50b8461186ed9ce34020338f8dd2d3c58c284b55"}
Nov 03 00:22:06.778454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:06.778001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:06.923655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:06.923620    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:22:06.923828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:06.923688    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:22:07.499075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.499045    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:07.554889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.554806    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:07.627031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.626996    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca\") pod \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") "
Nov 03 00:22:07.627212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.627066    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9\" (UniqueName: \"kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9\") pod \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") "
Nov 03 00:22:07.627212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.627118    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca\") pod \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\" (UID: \"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1\") "
Nov 03 00:22:07.627498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.627464    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca" pod "a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" (UID: "a8e5c23d-389c-44b3-83b3-3264e8d0ffa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:22:07.627595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.627558    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:22:07.629384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.629356    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca" pod "a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" (UID: "a8e5c23d-389c-44b3-83b3-3264e8d0ffa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:22:07.629522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.629379    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9" pod "a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" (UID: "a8e5c23d-389c-44b3-83b3-3264e8d0ffa1"). InnerVolumeSpecName "kube-api-access-4bkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:22:07.729228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.729186    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:22:07.729228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.729220    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkb9\" (UniqueName: \"kubernetes.io/projected/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1-kube-api-access-4bkb9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:22:07.923586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.923549    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:22:07.923800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:07.923611    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:22:08.304411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.304377    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"
Nov 03 00:22:08.304411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.304391    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl" event={"ID":"a8e5c23d-389c-44b3-83b3-3264e8d0ffa1","Type":"ContainerDied","Data":"fe5157518198cfea18785eed796af5402dbe238dec5774274218d19eb42dff3d"}
Nov 03 00:22:08.304974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.304478    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5157518198cfea18785eed796af5402dbe238dec5774274218d19eb42dff3d"
Nov 03 00:22:08.511761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.511725    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"]
Nov 03 00:22:08.512960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.512933    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" containerName="revoke-keys"
Nov 03 00:22:08.513133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.513118    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" containerName="revoke-keys"
Nov 03 00:22:08.513750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.513422    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" containerName="revoke-keys"
Nov 03 00:22:08.519893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.519867    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"]
Nov 03 00:22:08.520120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.520007    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.535885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.535857    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9f76b708\"/\"kube-root-ca.crt\""
Nov 03 00:22:08.536091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.535886    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9f76b708\"/\"e2e-lifecycle-model-9f76b708-kserve-self-signed-certs\""
Nov 03 00:22:08.536193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.535950    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9f76b708\"/\"default-dockercfg-tx4zg\""
Nov 03 00:22:08.536363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.536348    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9f76b708\"/\"openshift-service-ca.crt\""
Nov 03 00:22:08.577857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.573776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:08.640023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.639977    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhfvr\" (UniqueName: \"kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.640234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.640051    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.640234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.640104    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.640234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.640176    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.640234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.640214    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.640486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.640236    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742002    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhfvr\" (UniqueName: \"kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742086    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742164    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742300    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742367    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742407    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.742936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.742636    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.743063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.743042    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.743250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.743227    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.745626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.745591    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.745871 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.745846    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.771457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.766545    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhfvr\" (UniqueName: \"kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr\") pod \"e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") " pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.836586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.835224    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:08.933216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.922961    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:22:08.933216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:08.923015    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:22:09.287480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.281887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:09.300135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.300109    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"]
Nov 03 00:22:09.315229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.315200    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" event={"ID":"8255effd-ebb8-4eee-9730-c41ae7c18ee7","Type":"ContainerStarted","Data":"6c7ea0f1fb74aaba6be48f573787e41d042dacfb56b6fa934e6b5cfd9037232c"}
Nov 03 00:22:09.595513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.594961    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:09.922968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.922938    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:22:09.923148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.922992    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:22:09.978109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:09.978070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:10.324466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:10.324366    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" event={"ID":"8255effd-ebb8-4eee-9730-c41ae7c18ee7","Type":"ContainerStarted","Data":"2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5"}
Nov 03 00:22:10.369107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:10.369039    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" podStartSLOduration=2.369020219 podStartE2EDuration="2.369020219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:10.363686087 +0000 UTC m=+1012.685222886" watchObservedRunningTime="2026-11-03 00:22:10.369020219 +0000 UTC m=+1012.690557022"
Nov 03 00:22:10.836202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:10.836167    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:10.923808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:10.923775    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:22:10.924021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:10.923836    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Nov 03 00:22:11.318891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:11.318855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:11.927957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:11.927927    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:11.928421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:11.928399    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:11.929218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:11.929200    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:22:13.010404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:13.010363    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:14.606566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:14.606525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:14.746323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:14.746277    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:16.291609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:16.291575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:16.415142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:16.415106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:17.956131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.956096    4550 scope.go:122] "RemoveContainer" containerID="23cfb6e60135b8dac98a49d53b97f2dc5aa33313666c6816cd63a2a877e1ea6a"
Nov 03 00:22:17.960778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.960746    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"]
Nov 03 00:22:17.965335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.965312    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:17.967780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.967760    4550 scope.go:122] "RemoveContainer" containerID="717327e6bd111647c1bffc3892f660ab711258e5e92771b63218598e35a5eebe"
Nov 03 00:22:17.980737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.980656    4550 scope.go:122] "RemoveContainer" containerID="af21a536ba4bd1187ae1ac239d04fe3d7ccb00e5b8f56a4891e46ce6bdd1289d"
Nov 03 00:22:17.983577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.983551    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"]
Nov 03 00:22:17.991983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.991952    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:22:17.992090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.991987    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:22:17.994019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:17.993999    4550 scope.go:122] "RemoveContainer" containerID="517ec9c41397a0c2a0cdc961e408d415ee8226908fbee9767c1d3e21eb81e268"
Nov 03 00:22:18.010079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.010043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.010231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.010114    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.010418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.010335    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkkbq\" (UniqueName: \"kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.112109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.112069    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkkbq\" (UniqueName: \"kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.112273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.112218    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.112273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.112264    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.112954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.112931    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.114740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.114713    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.132216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.132184    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkkbq\" (UniqueName: \"kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq\") pod \"maas-api-revoke-sub-e33b8b94-7frzw\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.137497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.137464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:18.288005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.287918    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:18.597507 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:22:18.597470    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1c9071_76a8_465f_9ca2_b9b3630128b5.slice/crio-80b6e8885d2e37af7dc495b88b7c8b1a7e31156d339ff2e8e666fe1f012de5e1 WatchSource:0}: Error finding container 80b6e8885d2e37af7dc495b88b7c8b1a7e31156d339ff2e8e666fe1f012de5e1: Status 404 returned error can't find the container with id 80b6e8885d2e37af7dc495b88b7c8b1a7e31156d339ff2e8e666fe1f012de5e1
Nov 03 00:22:18.599611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.599572    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"]
Nov 03 00:22:18.836456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.836410    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:18.850159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:18.850081    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:19.391744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:19.391708    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw" event={"ID":"6c1c9071-76a8-465f-9ca2-b9b3630128b5","Type":"ContainerStarted","Data":"cb3a9521e0e5ba4117e652c1dcab87b002b596f16b9dd42c6a87a4d69909d566"}
Nov 03 00:22:19.392200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:19.391763    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw" event={"ID":"6c1c9071-76a8-465f-9ca2-b9b3630128b5","Type":"ContainerStarted","Data":"80b6e8885d2e37af7dc495b88b7c8b1a7e31156d339ff2e8e666fe1f012de5e1"}
Nov 03 00:22:19.406392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:19.406365    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:19.434824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:19.434774    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw" podStartSLOduration=2.434760108 podStartE2EDuration="2.434760108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:19.43130876 +0000 UTC m=+1021.752845578" watchObservedRunningTime="2026-11-03 00:22:19.434760108 +0000 UTC m=+1021.756296907"
Nov 03 00:22:19.968894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:19.968848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:20.399644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:20.399551    4550 generic.go:356] "Generic (PLEG): container finished" podID="6c1c9071-76a8-465f-9ca2-b9b3630128b5" containerID="cb3a9521e0e5ba4117e652c1dcab87b002b596f16b9dd42c6a87a4d69909d566" exitCode=0
Nov 03 00:22:20.400058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:20.399639    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw" event={"ID":"6c1c9071-76a8-465f-9ca2-b9b3630128b5","Type":"ContainerDied","Data":"cb3a9521e0e5ba4117e652c1dcab87b002b596f16b9dd42c6a87a4d69909d566"}
Nov 03 00:22:20.761160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:20.761049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:21.562073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.562043    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:21.626584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.626539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:21.664205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.664107    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca\") pod \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") "
Nov 03 00:22:21.664500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.664475    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca\") pod \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") "
Nov 03 00:22:21.664846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.664818    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq\" (UniqueName: \"kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq\") pod \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\" (UID: \"6c1c9071-76a8-465f-9ca2-b9b3630128b5\") "
Nov 03 00:22:21.666467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.666423    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca" pod "6c1c9071-76a8-465f-9ca2-b9b3630128b5" (UID: "6c1c9071-76a8-465f-9ca2-b9b3630128b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:22:21.670018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.669969    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq" pod "6c1c9071-76a8-465f-9ca2-b9b3630128b5" (UID: "6c1c9071-76a8-465f-9ca2-b9b3630128b5"). InnerVolumeSpecName "kube-api-access-qkkbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:22:21.679144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.679090    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca" pod "6c1c9071-76a8-465f-9ca2-b9b3630128b5" (UID: "6c1c9071-76a8-465f-9ca2-b9b3630128b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:22:21.767005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.766957    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkkbq\" (UniqueName: \"kubernetes.io/projected/6c1c9071-76a8-465f-9ca2-b9b3630128b5-kube-api-access-qkkbq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:22:21.767005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.766998    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:22:21.767005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:21.767014    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c1c9071-76a8-465f-9ca2-b9b3630128b5-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:22:22.413417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:22.413388    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"
Nov 03 00:22:22.413595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:22.413425    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw" event={"ID":"6c1c9071-76a8-465f-9ca2-b9b3630128b5","Type":"ContainerDied","Data":"80b6e8885d2e37af7dc495b88b7c8b1a7e31156d339ff2e8e666fe1f012de5e1"}
Nov 03 00:22:22.413595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:22.413513    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b6e8885d2e37af7dc495b88b7c8b1a7e31156d339ff2e8e666fe1f012de5e1"
Nov 03 00:22:23.235942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:23.235907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:24.724943 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:22:24.724907    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:22:24.829151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:24.829114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:25.854419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:25.854383    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:25.954312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:25.954278    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:26.388947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:26.388878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:28.861455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:28.860009    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:29.651629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:29.651525    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.853109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:29.851929    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:22:30.067187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.066555    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.297760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.297712    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.418455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.405242    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded"
Nov 03 00:22:30.524747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.524684    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.594818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.594781    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:30.660454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.657689    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.660454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.659647    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:22:30.904805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.877926    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.906473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:30.906444    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:22:31.047563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.047512    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:31.075795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.075749    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:31.077480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.077456    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:22:31.105759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.105717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:31.203615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.203512    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:22:31.224801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.224751    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 00:22:31.232499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.232469    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:22:31.430696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.430665    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:22:31.450024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.449966    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:31.450877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.450851    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:31.678998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.678962    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:22:31.698141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.698110    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:22:31.753535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.751450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:31.899486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:31.898142    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:33.637937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:33.637884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:35.368927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:35.368891    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:36.106478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:36.106417    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:36.219503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:36.219427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:37.332499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:37.332466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:37.788041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:37.788004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:37.885550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:37.885496    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:39.119639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:39.119588    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:40.896282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:40.896243    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:42.499900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:42.499862    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:44.115053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:44.115018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:45.823662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:45.823625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:47.684588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:47.684549    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:47.777321 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:47.777262    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:47.888015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:47.887977    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:48.621422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.621376    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"]
Nov 03 00:22:48.622285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.622260    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6c1c9071-76a8-465f-9ca2-b9b3630128b5" containerName="revoke-keys"
Nov 03 00:22:48.622285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.622284    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1c9071-76a8-465f-9ca2-b9b3630128b5" containerName="revoke-keys"
Nov 03 00:22:48.622507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.622494    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1c9071-76a8-465f-9ca2-b9b3630128b5" containerName="revoke-keys"
Nov 03 00:22:48.626709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.626685    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.635853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.635824    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"]
Nov 03 00:22:48.640300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.640278    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.659845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.659817    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"]
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712296    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsk5\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712389    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712528    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712581    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712613    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712638    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.712697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712672    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712743    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712775    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712850    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712875    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6g9b\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712906    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712951    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712976    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.712998    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.713043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.713069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.714991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.713091    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.720732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.720708    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"]
Nov 03 00:22:48.727175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.727151    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"]
Nov 03 00:22:48.727371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.727359    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"]
Nov 03 00:22:48.727543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.727519    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.817508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.817202    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.817508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.817251    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.817508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.817278    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.817508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.817309    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.817508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.817340    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsk5\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.817508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.817369    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820086    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820156    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820193    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820219    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820237    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820258    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820275    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820290    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrlln\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820318    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820353    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820404    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820449    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820478    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820505    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820638    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6g9b\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820680    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820731    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820798    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820833    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.820864    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.821726    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.822534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.821996    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.824267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.824245    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.824756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.824733    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.824970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.824943    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.825222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.825205    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.825325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.825205    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.825695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.825673    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.826955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.826924    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.827315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.827195    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.829416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.829374    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.831837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.831787    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.832945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.832905    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.833632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.833587    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.844599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.844118    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.845257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.845196    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.847326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.846378    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsk5\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.847326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.846734    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6g9b\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:48.922730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.922694    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.922951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.922749    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.922951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.922778    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrlln\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.922951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.922828    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.922951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.922900    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.922951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.922921    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923015    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923083    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923100    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923117    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923347    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923397    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923511    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.923943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.923921    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.925941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.925871    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.926167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.926129    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.943356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.943331    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.943528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.943506    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrlln\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:48.991230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.991199    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:48.994358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:48.994329    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:49.054573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.053797    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:49.347988 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:22:49.347946    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10124a4_efdf_408b_9602_6b9ff745217d.slice/crio-b47e19e0157d30cacb1247fbbb3797f3d92f61dec1a6b47640d928eb1b101a8c WatchSource:0}: Error finding container b47e19e0157d30cacb1247fbbb3797f3d92f61dec1a6b47640d928eb1b101a8c: Status 404 returned error can't find the container with id b47e19e0157d30cacb1247fbbb3797f3d92f61dec1a6b47640d928eb1b101a8c
Nov 03 00:22:49.350272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.350251    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"]
Nov 03 00:22:49.350503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.350471    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.350578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.350543    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.350578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.350571    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.368947 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:22:49.368911    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfdf551_5504_4e4b_8a40_954334350eb0.slice/crio-9655dc6db1717e9ec4152fb1c461296fb0b924e6c8d9ddfb74917b4298139f87 WatchSource:0}: Error finding container 9655dc6db1717e9ec4152fb1c461296fb0b924e6c8d9ddfb74917b4298139f87: Status 404 returned error can't find the container with id 9655dc6db1717e9ec4152fb1c461296fb0b924e6c8d9ddfb74917b4298139f87
Nov 03 00:22:49.371083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.371057    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"]
Nov 03 00:22:49.371665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.371632    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.371735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.371717    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.371781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.371766    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.466129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.466088    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"]
Nov 03 00:22:49.468949 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:22:49.468916    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd539aa_4521_49cc_9c0a_69fc213edceb.slice/crio-2a6ae3920e6f6a04b00cf8cf1e77039920e67205c9bfc02975f5f9fe99c4d044 WatchSource:0}: Error finding container 2a6ae3920e6f6a04b00cf8cf1e77039920e67205c9bfc02975f5f9fe99c4d044: Status 404 returned error can't find the container with id 2a6ae3920e6f6a04b00cf8cf1e77039920e67205c9bfc02975f5f9fe99c4d044
Nov 03 00:22:49.474636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.472124    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.474636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.472205    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.474636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.472251    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:22:49.624964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.624893    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:49.834744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.833489    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" event={"ID":"b10124a4-efdf-408b-9602-6b9ff745217d","Type":"ContainerStarted","Data":"a6d3249a010a053a863d267bce2abdfef683e0ffb3dccec815180c73894d7705"}
Nov 03 00:22:49.834744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.833567    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" event={"ID":"b10124a4-efdf-408b-9602-6b9ff745217d","Type":"ContainerStarted","Data":"b47e19e0157d30cacb1247fbbb3797f3d92f61dec1a6b47640d928eb1b101a8c"}
Nov 03 00:22:49.837638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.835850    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" event={"ID":"0bfdf551-5504-4e4b-8a40-954334350eb0","Type":"ContainerStarted","Data":"16904ce90ac94dc33383850e3a8eb1b0dd588595416668f16915bc6d1351f192"}
Nov 03 00:22:49.837638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.835909    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" event={"ID":"0bfdf551-5504-4e4b-8a40-954334350eb0","Type":"ContainerStarted","Data":"9655dc6db1717e9ec4152fb1c461296fb0b924e6c8d9ddfb74917b4298139f87"}
Nov 03 00:22:49.838900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.838873    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" event={"ID":"2dd539aa-4521-49cc-9c0a-69fc213edceb","Type":"ContainerStarted","Data":"897e1d3c25bd295e5a4c695279c494611a15dba3c90304dcbc9535cd17681306"}
Nov 03 00:22:49.839076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.839057    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" event={"ID":"2dd539aa-4521-49cc-9c0a-69fc213edceb","Type":"ContainerStarted","Data":"2a6ae3920e6f6a04b00cf8cf1e77039920e67205c9bfc02975f5f9fe99c4d044"}
Nov 03 00:22:49.883176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.882921    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podStartSLOduration=1.882901143 podStartE2EDuration="1.882901143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:49.876362848 +0000 UTC m=+1052.197899687" watchObservedRunningTime="2026-11-03 00:22:49.882901143 +0000 UTC m=+1052.204437955"
Nov 03 00:22:49.928081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.928010    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podStartSLOduration=1.927990341 podStartE2EDuration="1.927990341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:49.924943024 +0000 UTC m=+1052.246479835" watchObservedRunningTime="2026-11-03 00:22:49.927990341 +0000 UTC m=+1052.249527146"
Nov 03 00:22:49.984456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:49.983910    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podStartSLOduration=1.983885908 podStartE2EDuration="1.983885908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:49.965880737 +0000 UTC m=+1052.287417564" watchObservedRunningTime="2026-11-03 00:22:49.983885908 +0000 UTC m=+1052.305422709"
Nov 03 00:22:50.005835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.004806    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:50.005835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.005091    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:50.010269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.010228    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:50.010425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.010294    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:50.010571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.010242    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:50.010634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.010591    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:50.055109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.055037    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:50.060464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.059535    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:50.060464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:50.059596    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:51.037456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.030064    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.037456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.030127    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:51.053575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.049578    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.053575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.049646    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:51.067460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.063538    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.067460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.063600    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:51.610093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.610003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:51.996514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.996461    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.996752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.996549    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:51.996860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.996837    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.997002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:51.996978    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:52.075458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:52.058221    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:52.075458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:52.058284    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:53.002455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.002233    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:53.002455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.002360    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:53.002788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.002279    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:53.002933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.002907    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:53.054833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.054798    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:53.055100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.055078    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:53.548644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:53.548589    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:54.007460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.004519    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.007460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.004583    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:54.007460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.005000    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.007460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.005036    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:54.054634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.054598    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.054790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.054654    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:54.817982 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:22:54.817912    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:22:54.992458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.991883    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.992458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:54.991949    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:55.023456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:55.018784    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:55.023456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:55.018842    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:55.054980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:55.054947    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:55.061993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:55.055230    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:55.534281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:55.534236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:56.002949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:56.002912    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:56.003644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:56.003615    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Nov 03 00:22:56.004191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:56.003544    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:56.004343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:56.004324    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Nov 03 00:22:56.055702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:56.055663    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:22:56.055891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:56.055728    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 00:22:57.349027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.348881    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:22:57.410323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.409811    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:57.411465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.411407    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:57.419485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.419424    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:22:57.540367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.536744    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:57.540367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.538073    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:57.540367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.540264    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:22:57.621121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.621044    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:57.621395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.621369    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:57.622978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:57.622956    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:22:58.948845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:22:58.948808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:00.610619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:00.610580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:01.038303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:01.038263    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"]
Nov 03 00:23:01.050845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:01.050807    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bcf952c-wpw6t"]
Nov 03 00:23:01.809154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:01.809123    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3940b3-bdf1-4ca2-9955-fd60a8d74340" path="/var/lib/kubelet/pods/dc3940b3-bdf1-4ca2-9955-fd60a8d74340/volumes"
Nov 03 00:23:02.332625 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:02.332591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:04.372882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:04.372832    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:06.248365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:06.248322    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:06.422287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:06.422248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:07.117359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.117324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:07.505454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.497920    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"]
Nov 03 00:23:07.515673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.515642    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"]
Nov 03 00:23:07.516011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.515952    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.529087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.528016    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:07.533919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.532738    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-714512-openshift-default-dockercfg-kk7d9\""
Nov 03 00:23:07.558322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558272    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558333    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558481    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558524    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbgb\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558563    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558630    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558742    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.558910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.558847    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.660373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.660326    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.662182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.662153    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.662800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.662767    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.663380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.663348    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbgb\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.664010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.663981    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.665277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.661049    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.674500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.665162    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.675090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.675064    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.675711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.675692    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.676382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.676364    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.686721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.686693    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.687068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.686145    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.687182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.676787    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.687599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.687577    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.687799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.675522    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.704966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.704220    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.718723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.711176    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbgb\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.719392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.719322    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy\") pod \"e2e-api-b-714512-openshift-default-8fb84d547-nv6vg\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") " pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:07.833525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.833397    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:07.853150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:07.852510    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:08.234540 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:23:08.234424    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcd8a18_6b88_449a_a32e_9fd43b678269.slice/crio-d4729fb53aa1bd7a3e662611d1727c76c4b6cbfe20c889c1f1ad7003d27bfbed WatchSource:0}: Error finding container d4729fb53aa1bd7a3e662611d1727c76c4b6cbfe20c889c1f1ad7003d27bfbed: Status 404 returned error can't find the container with id d4729fb53aa1bd7a3e662611d1727c76c4b6cbfe20c889c1f1ad7003d27bfbed
Nov 03 00:23:08.236451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.236393    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"]
Nov 03 00:23:08.237642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.237610    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:08.237768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.237690    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:08.237768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.237739    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:23:08.328744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.328712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:08.357694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.357642    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" event={"ID":"ddcd8a18-6b88-449a-a32e-9fd43b678269","Type":"ContainerStarted","Data":"561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"}
Nov 03 00:23:08.357854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.357751    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" event={"ID":"ddcd8a18-6b88-449a-a32e-9fd43b678269","Type":"ContainerStarted","Data":"d4729fb53aa1bd7a3e662611d1727c76c4b6cbfe20c889c1f1ad7003d27bfbed"}
Nov 03 00:23:08.411167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.411084    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podStartSLOduration=1.411050153 podStartE2EDuration="1.411050153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:08.400926834 +0000 UTC m=+1070.722463647" watchObservedRunningTime="2026-11-03 00:23:08.411050153 +0000 UTC m=+1070.732586954"
Nov 03 00:23:08.854515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:08.854480    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:09.385525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:09.385488    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:09.861568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:09.861537    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:10.061849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:10.061816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:10.371984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:10.371903    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:10.372936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:10.372912    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:23:10.766252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:10.766221    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:10.893365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:10.893329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:11.931233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:11.931192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:12.516013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:12.515976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:12.613820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:12.613782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:13.952034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:13.952006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:14.227612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.227521    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7618-e9313b"]
Nov 03 00:23:14.231606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.231586    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:14.248145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.248116    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7618-e9313b"]
Nov 03 00:23:14.257502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.257478    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 00:23:14.406390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.406352    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pxb\" (UniqueName: \"kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb\") pod \"test-curl-7618-e9313b\" (UID: \"fa71090e-142c-4add-9dfb-9842e8b24e36\") " pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:14.508275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.508186    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pxb\" (UniqueName: \"kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb\") pod \"test-curl-7618-e9313b\" (UID: \"fa71090e-142c-4add-9dfb-9842e8b24e36\") " pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:14.528474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.528444    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pxb\" (UniqueName: \"kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb\") pod \"test-curl-7618-e9313b\" (UID: \"fa71090e-142c-4add-9dfb-9842e8b24e36\") " pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:14.544601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.544567    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:14.843215 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:23:14.843178    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa71090e_142c_4add_9dfb_9842e8b24e36.slice/crio-f8d3ff29870677ac265c4a464fb3d037c7e6f284b9f8cb9fc5152d695257baf8 WatchSource:0}: Error finding container f8d3ff29870677ac265c4a464fb3d037c7e6f284b9f8cb9fc5152d695257baf8: Status 404 returned error can't find the container with id f8d3ff29870677ac265c4a464fb3d037c7e6f284b9f8cb9fc5152d695257baf8
Nov 03 00:23:14.845142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.845126    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7618-e9313b"]
Nov 03 00:23:14.845601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:14.845586    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:23:15.030174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:15.030138    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"]
Nov 03 00:23:15.043328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:15.043296    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d5bccf-cfvnb"]
Nov 03 00:23:15.421160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:15.421123    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7618-e9313b" event={"ID":"fa71090e-142c-4add-9dfb-9842e8b24e36","Type":"ContainerStarted","Data":"f8d3ff29870677ac265c4a464fb3d037c7e6f284b9f8cb9fc5152d695257baf8"}
Nov 03 00:23:15.753562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:15.753470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:15.810445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:15.810403    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541ffad3-bf52-4b93-ba74-0913eb5be389" path="/var/lib/kubelet/pods/541ffad3-bf52-4b93-ba74-0913eb5be389/volumes"
Nov 03 00:23:17.436405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:17.436376    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7618-e9313b" event={"ID":"fa71090e-142c-4add-9dfb-9842e8b24e36","Type":"ContainerStarted","Data":"25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256"}
Nov 03 00:23:17.448839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:17.448812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:17.477113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:17.477059    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7618-e9313b" podStartSLOduration=0.984219955 podStartE2EDuration="3.477047481s" totalImagesPullingTime="2.492827526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:17.474408195 +0000 UTC m=+1079.795944997" watchObservedRunningTime="2026-11-03 00:23:17.477047481 +0000 UTC m=+1079.798584280"
Nov 03 00:23:18.336387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:18.336353    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7618-e9313b"]
Nov 03 00:23:18.342456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:18.342399    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7618-e9313b"]
Nov 03 00:23:19.136362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.136325    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"]
Nov 03 00:23:19.136915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.136900    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fa71090e-142c-4add-9dfb-9842e8b24e36" containerName="test-curl-7618-e9313b"
Nov 03 00:23:19.136961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.136916    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa71090e-142c-4add-9dfb-9842e8b24e36" containerName="test-curl-7618-e9313b"
Nov 03 00:23:19.137036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.137025    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa71090e-142c-4add-9dfb-9842e8b24e36" containerName="test-curl-7618-e9313b"
Nov 03 00:23:19.141206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.141189    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.167922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.167890    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:23:19.168101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.168014    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:23:19.169568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.169535    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"]
Nov 03 00:23:19.174314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.174283    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.174592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.174569    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbc8\" (UniqueName: \"kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.174649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.174619    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.276150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.276116    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbc8\" (UniqueName: \"kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.276371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.276156    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.276371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.276249    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.277012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.276985    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.278715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.278686    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.296796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.296764    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbc8\" (UniqueName: \"kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8\") pod \"maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.315806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.315769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:19.451831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.451578    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7618-e9313b" podUID="fa71090e-142c-4add-9dfb-9842e8b24e36" containerName="test-curl-7618-e9313b" containerID="cri-o://25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256" gracePeriod=2
Nov 03 00:23:19.454796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.454766    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:19.467780 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:23:19.465883    4550 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-e9313b\" is forbidden: User \"system:node:ip-10-0-133-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-253.ec2.internal' and this object" podUID="fa71090e-142c-4add-9dfb-9842e8b24e36" pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:19.768185 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:23:19.768151    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d04edce_9ced_4096_8b1a_6f82fac637c2.slice/crio-116d5ca3df361c099c96c0350a240ba643dfec2d5cf2939de0bfdf9b23ba5add WatchSource:0}: Error finding container 116d5ca3df361c099c96c0350a240ba643dfec2d5cf2939de0bfdf9b23ba5add: Status 404 returned error can't find the container with id 116d5ca3df361c099c96c0350a240ba643dfec2d5cf2939de0bfdf9b23ba5add
Nov 03 00:23:19.769691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:19.769662    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"]
Nov 03 00:23:20.460636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:20.460600    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn" event={"ID":"7d04edce-9ced-4096-8b1a-6f82fac637c2","Type":"ContainerStarted","Data":"bac6ec73ed079827afb2d33b805d4bda97949c33d2f9ef4eb6029cbb016d10ec"}
Nov 03 00:23:20.461164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:20.460657    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn" event={"ID":"7d04edce-9ced-4096-8b1a-6f82fac637c2","Type":"ContainerStarted","Data":"116d5ca3df361c099c96c0350a240ba643dfec2d5cf2939de0bfdf9b23ba5add"}
Nov 03 00:23:20.502246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:20.502176    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn" podStartSLOduration=1.5021561860000001 podStartE2EDuration="1.502156186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:20.499104241 +0000 UTC m=+1082.820641042" watchObservedRunningTime="2026-11-03 00:23:20.502156186 +0000 UTC m=+1082.823692991"
Nov 03 00:23:21.129126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.129086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:21.469481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.469411    4550 generic.go:356] "Generic (PLEG): container finished" podID="7d04edce-9ced-4096-8b1a-6f82fac637c2" containerID="bac6ec73ed079827afb2d33b805d4bda97949c33d2f9ef4eb6029cbb016d10ec" exitCode=0
Nov 03 00:23:21.469820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.469496    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn" event={"ID":"7d04edce-9ced-4096-8b1a-6f82fac637c2","Type":"ContainerDied","Data":"bac6ec73ed079827afb2d33b805d4bda97949c33d2f9ef4eb6029cbb016d10ec"}
Nov 03 00:23:21.660863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.660838    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:21.710684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.710603    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb\" (UniqueName: \"kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb\") pod \"fa71090e-142c-4add-9dfb-9842e8b24e36\" (UID: \"fa71090e-142c-4add-9dfb-9842e8b24e36\") "
Nov 03 00:23:21.712776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.712744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb" pod "fa71090e-142c-4add-9dfb-9842e8b24e36" (UID: "fa71090e-142c-4add-9dfb-9842e8b24e36"). InnerVolumeSpecName "kube-api-access-j7pxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:21.807888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.807852    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa71090e-142c-4add-9dfb-9842e8b24e36" path="/var/lib/kubelet/pods/fa71090e-142c-4add-9dfb-9842e8b24e36/volumes"
Nov 03 00:23:21.812042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:21.812011    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7pxb\" (UniqueName: \"kubernetes.io/projected/fa71090e-142c-4add-9dfb-9842e8b24e36-kube-api-access-j7pxb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:22.478279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.478239    4550 generic.go:356] "Generic (PLEG): container finished" podID="fa71090e-142c-4add-9dfb-9842e8b24e36" containerID="25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256" exitCode=137
Nov 03 00:23:22.478854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.478290    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-e9313b"
Nov 03 00:23:22.478854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.478341    4550 scope.go:122] "RemoveContainer" containerID="25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256"
Nov 03 00:23:22.489444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.489409    4550 scope.go:122] "RemoveContainer" containerID="25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256"
Nov 03 00:23:22.489788 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:23:22.489762    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256\": container with ID starting with 25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256 not found: ID does not exist" containerID="25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256"
Nov 03 00:23:22.489856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.489800    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256"} err="failed to get container status \"25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256\": rpc error: code = NotFound desc = could not find container \"25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256\": container with ID starting with 25c84119ab72fd2ff9b73bff9ec36c4b450c557bd9c2eb3295d011cbbafb1256 not found: ID does not exist"
Nov 03 00:23:22.647686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.647660    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:22.725648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.725610    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca\") pod \"7d04edce-9ced-4096-8b1a-6f82fac637c2\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") "
Nov 03 00:23:22.725856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.725838    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8\" (UniqueName: \"kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8\") pod \"7d04edce-9ced-4096-8b1a-6f82fac637c2\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") "
Nov 03 00:23:22.725936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.725909    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca\") pod \"7d04edce-9ced-4096-8b1a-6f82fac637c2\" (UID: \"7d04edce-9ced-4096-8b1a-6f82fac637c2\") "
Nov 03 00:23:22.726343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.726313    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca" pod "7d04edce-9ced-4096-8b1a-6f82fac637c2" (UID: "7d04edce-9ced-4096-8b1a-6f82fac637c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:22.726611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.726589    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:22.728382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.728309    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca" pod "7d04edce-9ced-4096-8b1a-6f82fac637c2" (UID: "7d04edce-9ced-4096-8b1a-6f82fac637c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:23:22.728631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.728607    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8" pod "7d04edce-9ced-4096-8b1a-6f82fac637c2" (UID: "7d04edce-9ced-4096-8b1a-6f82fac637c2"). InnerVolumeSpecName "kube-api-access-kdbc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:22.828145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.828108    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d04edce-9ced-4096-8b1a-6f82fac637c2-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:22.828145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:22.828141    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdbc8\" (UniqueName: \"kubernetes.io/projected/7d04edce-9ced-4096-8b1a-6f82fac637c2-kube-api-access-kdbc8\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:23.021974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:23.021877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:23.488502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:23.485919    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"
Nov 03 00:23:23.488502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:23.485926    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn" event={"ID":"7d04edce-9ced-4096-8b1a-6f82fac637c2","Type":"ContainerDied","Data":"116d5ca3df361c099c96c0350a240ba643dfec2d5cf2939de0bfdf9b23ba5add"}
Nov 03 00:23:23.488502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:23.485981    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="116d5ca3df361c099c96c0350a240ba643dfec2d5cf2939de0bfdf9b23ba5add"
Nov 03 00:23:24.688998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:24.688956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:24.726475 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:23:24.726402    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:23:24.788104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:24.788060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:26.416844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:26.416802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:28.128174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:28.128127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:29.029217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:29.029184    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"]
Nov 03 00:23:29.044239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:29.044201    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7af7c9c6-wgzkb"]
Nov 03 00:23:29.690789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:29.690698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:29.788341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:29.788305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:29.807926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:29.807896    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0edebcc-2e4f-4263-b4db-fd9624270794" path="/var/lib/kubelet/pods/e0edebcc-2e4f-4263-b4db-fd9624270794/volumes"
Nov 03 00:23:29.889186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:29.889152    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:31.509942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:31.509906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:33.210579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:33.210535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:34.920800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:34.920768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:36.533574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:36.533535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:38.185645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:38.185609    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:39.861257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:39.861216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:40.842000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:40.841969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:41.276516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.276482    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"]
Nov 03 00:23:41.277330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.277302    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7d04edce-9ced-4096-8b1a-6f82fac637c2" containerName="revoke-keys"
Nov 03 00:23:41.277532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.277507    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d04edce-9ced-4096-8b1a-6f82fac637c2" containerName="revoke-keys"
Nov 03 00:23:41.277698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.277685    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d04edce-9ced-4096-8b1a-6f82fac637c2" containerName="revoke-keys"
Nov 03 00:23:41.281750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.281727    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"]
Nov 03 00:23:41.281891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.281869    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.289714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.289686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:41.309071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.309048    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:23:41.309325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.309301    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:23:41.387666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.387622    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:41.402002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.401968    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.402175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.402010    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclvm\" (UniqueName: \"kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.402310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.402282    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.510018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.508488    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.510018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.508538    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclvm\" (UniqueName: \"kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.510018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.508698    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.510018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.509607    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.516133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.516079    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.531546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.531449    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclvm\" (UniqueName: \"kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm\") pod \"maas-api-revoke-sub-7a2ed4e7-6npd2\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.603995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.603704    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:41.689324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.689254    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:41.992777 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:23:41.992072    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e69467e_9fb6_4734_aa70_26060ad42d31.slice/crio-786c8ac166f149c38c2ee7b7bf48360fb4d2bd2b770ff855247d42803d14e00d WatchSource:0}: Error finding container 786c8ac166f149c38c2ee7b7bf48360fb4d2bd2b770ff855247d42803d14e00d: Status 404 returned error can't find the container with id 786c8ac166f149c38c2ee7b7bf48360fb4d2bd2b770ff855247d42803d14e00d
Nov 03 00:23:41.995034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:41.995005    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"]
Nov 03 00:23:42.661661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:42.661621    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2" event={"ID":"0e69467e-9fb6-4734-aa70-26060ad42d31","Type":"ContainerStarted","Data":"264209fb4f00f670f787389aa03856ab25d688544744cefb32ad2340f198f9c5"}
Nov 03 00:23:42.662126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:42.661687    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2" event={"ID":"0e69467e-9fb6-4734-aa70-26060ad42d31","Type":"ContainerStarted","Data":"786c8ac166f149c38c2ee7b7bf48360fb4d2bd2b770ff855247d42803d14e00d"}
Nov 03 00:23:42.701767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:42.701714    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2" podStartSLOduration=1.701699606 podStartE2EDuration="1.701699606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:42.700495248 +0000 UTC m=+1105.022032047" watchObservedRunningTime="2026-11-03 00:23:42.701699606 +0000 UTC m=+1105.023236406"
Nov 03 00:23:43.458857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:43.458814    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:43.670685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:43.670653    4550 generic.go:356] "Generic (PLEG): container finished" podID="0e69467e-9fb6-4734-aa70-26060ad42d31" containerID="264209fb4f00f670f787389aa03856ab25d688544744cefb32ad2340f198f9c5" exitCode=0
Nov 03 00:23:43.671157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:43.670745    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2" event={"ID":"0e69467e-9fb6-4734-aa70-26060ad42d31","Type":"ContainerDied","Data":"264209fb4f00f670f787389aa03856ab25d688544744cefb32ad2340f198f9c5"}
Nov 03 00:23:44.828627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.828601    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:44.965443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.965348    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca\") pod \"0e69467e-9fb6-4734-aa70-26060ad42d31\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") "
Nov 03 00:23:44.965606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.965506    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm\" (UniqueName: \"kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm\") pod \"0e69467e-9fb6-4734-aa70-26060ad42d31\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") "
Nov 03 00:23:44.965606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.965591    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca\") pod \"0e69467e-9fb6-4734-aa70-26060ad42d31\" (UID: \"0e69467e-9fb6-4734-aa70-26060ad42d31\") "
Nov 03 00:23:44.965852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.965821    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca" pod "0e69467e-9fb6-4734-aa70-26060ad42d31" (UID: "0e69467e-9fb6-4734-aa70-26060ad42d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:44.966129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.966110    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:44.967685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.967661    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca" pod "0e69467e-9fb6-4734-aa70-26060ad42d31" (UID: "0e69467e-9fb6-4734-aa70-26060ad42d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:23:44.968081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:44.968053    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm" pod "0e69467e-9fb6-4734-aa70-26060ad42d31" (UID: "0e69467e-9fb6-4734-aa70-26060ad42d31"). InnerVolumeSpecName "kube-api-access-vclvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:45.067138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.067100    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vclvm\" (UniqueName: \"kubernetes.io/projected/0e69467e-9fb6-4734-aa70-26060ad42d31-kube-api-access-vclvm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:45.067138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.067130    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e69467e-9fb6-4734-aa70-26060ad42d31-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:45.249827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.249743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:45.504522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.504419    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"]
Nov 03 00:23:45.505189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.505173    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0e69467e-9fb6-4734-aa70-26060ad42d31" containerName="revoke-keys"
Nov 03 00:23:45.505273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.505191    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e69467e-9fb6-4734-aa70-26060ad42d31" containerName="revoke-keys"
Nov 03 00:23:45.505393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.505381    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e69467e-9fb6-4734-aa70-26060ad42d31" containerName="revoke-keys"
Nov 03 00:23:45.509180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.509157    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.512989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.512963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:45.528212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.528184    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"]
Nov 03 00:23:45.574891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.574858    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.575064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.574901    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.575064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.574928    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqb2x\" (UniqueName: \"kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.625080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.625047    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:45.676652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.676609    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.676864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.676659    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.676864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.676682    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqb2x\" (UniqueName: \"kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.677499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.677409    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.679478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.679449    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.686109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.686073    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2" event={"ID":"0e69467e-9fb6-4734-aa70-26060ad42d31","Type":"ContainerDied","Data":"786c8ac166f149c38c2ee7b7bf48360fb4d2bd2b770ff855247d42803d14e00d"}
Nov 03 00:23:45.686220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.686144    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="786c8ac166f149c38c2ee7b7bf48360fb4d2bd2b770ff855247d42803d14e00d"
Nov 03 00:23:45.686220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.686092    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"
Nov 03 00:23:45.699610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.699582    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqb2x\" (UniqueName: \"kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x\") pod \"maas-api-revoke-sub-90ca748b-pxg5k\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:45.824544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:45.824468    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:46.125139 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:23:46.125106    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15f42e6_cc9b_4524_99c0_a2552aad1d28.slice/crio-44c5edb5c67c38206235fc348c179912f379a15b478d2bcca20754b940a54357 WatchSource:0}: Error finding container 44c5edb5c67c38206235fc348c179912f379a15b478d2bcca20754b940a54357: Status 404 returned error can't find the container with id 44c5edb5c67c38206235fc348c179912f379a15b478d2bcca20754b940a54357
Nov 03 00:23:46.126977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:46.126950    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"]
Nov 03 00:23:46.335368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:46.335329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:46.695901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:46.695862    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k" event={"ID":"a15f42e6-cc9b-4524-99c0-a2552aad1d28","Type":"ContainerStarted","Data":"2dc19b4c23af9deabaa0daaa75362c6cb51321b6bac84bc01c0e1e344ac5bb3d"}
Nov 03 00:23:46.696122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:46.695930    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k" event={"ID":"a15f42e6-cc9b-4524-99c0-a2552aad1d28","Type":"ContainerStarted","Data":"44c5edb5c67c38206235fc348c179912f379a15b478d2bcca20754b940a54357"}
Nov 03 00:23:46.740018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:46.739965    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k" podStartSLOduration=1.739951999 podStartE2EDuration="1.739951999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:46.736370563 +0000 UTC m=+1109.057907364" watchObservedRunningTime="2026-11-03 00:23:46.739951999 +0000 UTC m=+1109.061488811"
Nov 03 00:23:47.151283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:47.151192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:47.335328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:47.335170    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:47.481455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:47.464614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:47.715949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:47.715919    4550 generic.go:356] "Generic (PLEG): container finished" podID="a15f42e6-cc9b-4524-99c0-a2552aad1d28" containerID="2dc19b4c23af9deabaa0daaa75362c6cb51321b6bac84bc01c0e1e344ac5bb3d" exitCode=0
Nov 03 00:23:47.716312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:47.716288    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k" event={"ID":"a15f42e6-cc9b-4524-99c0-a2552aad1d28","Type":"ContainerDied","Data":"2dc19b4c23af9deabaa0daaa75362c6cb51321b6bac84bc01c0e1e344ac5bb3d"}
Nov 03 00:23:47.943565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:47.943532    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:48.915143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:48.915110    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:49.056317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.056215    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca\") pod \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") "
Nov 03 00:23:49.056317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.056307    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x\" (UniqueName: \"kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x\") pod \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") "
Nov 03 00:23:49.056597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.056343    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca\") pod \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\" (UID: \"a15f42e6-cc9b-4524-99c0-a2552aad1d28\") "
Nov 03 00:23:49.056661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.056637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca" pod "a15f42e6-cc9b-4524-99c0-a2552aad1d28" (UID: "a15f42e6-cc9b-4524-99c0-a2552aad1d28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:49.056916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.056897    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:49.058894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.058867    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca" pod "a15f42e6-cc9b-4524-99c0-a2552aad1d28" (UID: "a15f42e6-cc9b-4524-99c0-a2552aad1d28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:23:49.058894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.058881    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x" pod "a15f42e6-cc9b-4524-99c0-a2552aad1d28" (UID: "a15f42e6-cc9b-4524-99c0-a2552aad1d28"). InnerVolumeSpecName "kube-api-access-hqb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:49.157937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.157901    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqb2x\" (UniqueName: \"kubernetes.io/projected/a15f42e6-cc9b-4524-99c0-a2552aad1d28-kube-api-access-hqb2x\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:49.157937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.157934    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15f42e6-cc9b-4524-99c0-a2552aad1d28-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:23:49.222299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.222186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:49.731575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.731536    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k" event={"ID":"a15f42e6-cc9b-4524-99c0-a2552aad1d28","Type":"ContainerDied","Data":"44c5edb5c67c38206235fc348c179912f379a15b478d2bcca20754b940a54357"}
Nov 03 00:23:49.731575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.731573    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"
Nov 03 00:23:49.731790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:49.731620    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44c5edb5c67c38206235fc348c179912f379a15b478d2bcca20754b940a54357"
Nov 03 00:23:50.828501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:50.828464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:52.401012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:52.400976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:53.879721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:53.879685    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:54.018611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:54.018577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:54.118565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:54.118533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:54.275722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:54.275681    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:54.731180 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:23:54.731134    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:23:55.814717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:55.814679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:56.926826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:56.926774    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"]
Nov 03 00:23:56.940587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:56.940552    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9f76b708-9j2vn"]
Nov 03 00:23:57.417994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:57.417955    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:23:57.810323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:57.810194    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d04edce-9ced-4096-8b1a-6f82fac637c2" path="/var/lib/kubelet/pods/7d04edce-9ced-4096-8b1a-6f82fac637c2/volumes"
Nov 03 00:23:59.082080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:23:59.082044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:00.317402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:00.317323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:00.696868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:00.696832    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:00.788558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:00.788515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:00.899973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:00.899942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:01.050242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:01.050165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:01.933603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:01.933565    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:02.622884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:02.622835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:04.181701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:04.181646    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:04.433746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:04.433656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:04.529380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:04.529268    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:05.136945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.136914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:05.416533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.416503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:05.516142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.516037    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:05.873291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.873197    4550 generic.go:356] "Generic (PLEG): container finished" podID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerID="561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b" exitCode=0
Nov 03 00:24:05.873291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.873269    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" event={"ID":"ddcd8a18-6b88-449a-a32e-9fd43b678269","Type":"ContainerDied","Data":"561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"}
Nov 03 00:24:05.873747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.873731    4550 scope.go:122] "RemoveContainer" containerID="561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"
Nov 03 00:24:05.875678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.875634    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:05.875790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.875760    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:05.875855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:05.875839    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:06.889956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:06.889918    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" event={"ID":"ddcd8a18-6b88-449a-a32e-9fd43b678269","Type":"ContainerStarted","Data":"0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d"}
Nov 03 00:24:07.008716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:07.008675    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:07.115280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:07.115244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:07.785858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:07.785826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:07.854765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:07.854725    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:24:07.863476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:07.860477    4550 patch_prober.go:29] interesting pod/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 00:24:07.863476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:07.860538    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 00:24:08.302456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:08.301398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:08.854749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:08.854720    4550 patch_prober.go:29] interesting pod/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 00:24:08.854925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:08.854766    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 00:24:09.045262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:09.045230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:09.147648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:09.147555    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:09.496707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:09.496674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:09.854153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:09.854071    4550 patch_prober.go:29] interesting pod/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 00:24:09.854153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:09.854131    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 00:24:10.375505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:10.374513    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:24:10.847336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:10.847288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:10.854216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:10.854190    4550 patch_prober.go:29] interesting pod/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 00:24:10.854317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:10.854251    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 00:24:11.854681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:11.854647    4550 patch_prober.go:29] interesting pod/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 03 00:24:11.855184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:11.854694    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 03 00:24:12.436480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:12.436444    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:12.541979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:12.541950    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:12.643340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:12.643304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:12.859048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:12.858975    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:24:12.860157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:12.860135    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:24:14.419393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:14.419349    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:14.920710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:14.920679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:15.032322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:15.032281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:15.396334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:15.396231    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:16.441046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:16.441003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:18.051698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:18.051653    4550 scope.go:122] "RemoveContainer" containerID="53907806108c33b98fe1622b027fccc80301de566fdc06181d9fa0d3778ed834"
Nov 03 00:24:18.068006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:18.067969    4550 scope.go:122] "RemoveContainer" containerID="932041b2e164740a9669a13c1b5f59f36da0e85c2d830a091a486865d955bb30"
Nov 03 00:24:18.136799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:18.136758    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:20.186455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:20.185962    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:20.752235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:20.752110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:21.400100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:21.400066    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:21.928597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:21.928167    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:22.080805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:22.080773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:22.216999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:22.216873    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:22.951922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:22.951879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:24.631301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:24.631268    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:24.732797 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:24:24.732759    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:24:26.215556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:26.215524    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:26.327859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:26.327827    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:26.626917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:26.626825    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:27.942630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:27.942595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:28.042858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:28.042826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:28.616896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:28.616858    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:29.675806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:29.675772    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:31.708360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:31.708327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:33.526874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.526837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:33.943059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.943020    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"]
Nov 03 00:24:33.943628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.943610    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a15f42e6-cc9b-4524-99c0-a2552aad1d28" containerName="revoke-keys"
Nov 03 00:24:33.943628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.943627    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15f42e6-cc9b-4524-99c0-a2552aad1d28" containerName="revoke-keys"
Nov 03 00:24:33.943759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.943713    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15f42e6-cc9b-4524-99c0-a2552aad1d28" containerName="revoke-keys"
Nov 03 00:24:33.949355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.949334    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:33.967693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.967659    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:24:33.967851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.967762    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:24:33.968095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.968077    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:24:33.973840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.973816    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"]
Nov 03 00:24:33.980649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:33.980616    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wclrs\""
Nov 03 00:24:34.007064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007025    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.007064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.007340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007140    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.007340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007183    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq47v\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.007340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007228    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.007562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007458    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.007562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.007488    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108536    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108571    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108644    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108666    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108707    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108756    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq47v\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.108841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.108804    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.109511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.109482    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.111298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.111262    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.111407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.111313    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.111552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.111530    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.111850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.111832    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.128785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.128759    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq47v\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.128886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.128832    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qqfrs\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.262851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.262766    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:34.577537 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:24:34.577493    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ee0c1d_6f20_4b99_a3e3_422108b84ce4.slice/crio-054a54098cc37e7705af3efeaeb9b3097003ecf049fe4a57bcfb5cbb314775e1 WatchSource:0}: Error finding container 054a54098cc37e7705af3efeaeb9b3097003ecf049fe4a57bcfb5cbb314775e1: Status 404 returned error can't find the container with id 054a54098cc37e7705af3efeaeb9b3097003ecf049fe4a57bcfb5cbb314775e1
Nov 03 00:24:34.578498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:34.578475    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"]
Nov 03 00:24:35.198151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:35.198115    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" event={"ID":"92ee0c1d-6f20-4b99-a3e3-422108b84ce4","Type":"ContainerStarted","Data":"054a54098cc37e7705af3efeaeb9b3097003ecf049fe4a57bcfb5cbb314775e1"}
Nov 03 00:24:35.211399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:35.211365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:37.001375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:37.001337    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:37.226657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:37.226621    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:37.331865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:37.331785    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:37.689294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:37.689255    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:37.689418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:37.689326    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:24:37.939139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:37.939051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:38.222851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.222768    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" event={"ID":"92ee0c1d-6f20-4b99-a3e3-422108b84ce4","Type":"ContainerStarted","Data":"e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13"}
Nov 03 00:24:38.223229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.222938    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:38.224768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.224741    4550 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qqfrs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:24:38.224895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.224787    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" podUID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:24:38.310454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.299550    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" podStartSLOduration=2.191173829 podStartE2EDuration="5.299528071s" totalImagesPullingTime="3.108354242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:38.287832839 +0000 UTC m=+1160.609369676" watchObservedRunningTime="2026-11-03 00:24:38.299528071 +0000 UTC m=+1160.621064872"
Nov 03 00:24:38.341881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.341829    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"]
Nov 03 00:24:38.342359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.342325    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" containerID="cri-o://0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d" gracePeriod=30
Nov 03 00:24:38.359260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:38.351932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:39.034029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:39.033993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:39.230622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:39.230594    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:24:40.476327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:40.476288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:40.726743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:40.726647    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:42.310018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:42.309985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:42.407153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:42.407113    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:42.861955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:42.861916    4550 patch_prober.go:29] interesting pod/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:24:42.862123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:42.861977    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:24:43.383038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:43.383004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:44.155493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.155464    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:24:44.211477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.211403    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.211670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.211505    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.211670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.211589    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.211811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.211741    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.211927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.211908    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.212010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.211967    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.212010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.212000    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.212115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.212031    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.212172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.212135    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb\") pod \"ddcd8a18-6b88-449a-a32e-9fd43b678269\" (UID: \"ddcd8a18-6b88-449a-a32e-9fd43b678269\") "
Nov 03 00:24:44.212771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.212713    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:44.213006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.212903    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:44.213441 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.213149    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:44.213441 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.213174    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:44.213441 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.213214    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:44.215756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.215727    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:24:44.215849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.215816    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "kube-api-access-8xbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:44.215849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.215840    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:44.216566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.216535    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy" pod "ddcd8a18-6b88-449a-a32e-9fd43b678269" (UID: "ddcd8a18-6b88-449a-a32e-9fd43b678269"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:44.291089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.290358    4550 generic.go:356] "Generic (PLEG): container finished" podID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerID="0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d" exitCode=0
Nov 03 00:24:44.291089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.290530    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" event={"ID":"ddcd8a18-6b88-449a-a32e-9fd43b678269","Type":"ContainerDied","Data":"0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d"}
Nov 03 00:24:44.291089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.290589    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg" event={"ID":"ddcd8a18-6b88-449a-a32e-9fd43b678269","Type":"ContainerDied","Data":"d4729fb53aa1bd7a3e662611d1727c76c4b6cbfe20c889c1f1ad7003d27bfbed"}
Nov 03 00:24:44.291089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.290619    4550 scope.go:122] "RemoveContainer" containerID="0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d"
Nov 03 00:24:44.291089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.290760    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"
Nov 03 00:24:44.313994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.313961    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbgb\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-kube-api-access-8xbgb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314000    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314015    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314029    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314043    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314056    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddcd8a18-6b88-449a-a32e-9fd43b678269-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314068    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314083    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.314188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.314097    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddcd8a18-6b88-449a-a32e-9fd43b678269-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:24:44.319390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.319367    4550 scope.go:122] "RemoveContainer" containerID="561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"
Nov 03 00:24:44.330361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.330331    4550 scope.go:122] "RemoveContainer" containerID="0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d"
Nov 03 00:24:44.332326 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:24:44.332300    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d\": container with ID starting with 0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d not found: ID does not exist" containerID="0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d"
Nov 03 00:24:44.332422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.332335    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d"} err="failed to get container status \"0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d\": rpc error: code = NotFound desc = could not find container \"0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d\": container with ID starting with 0c7ac59cd8323915e4dfa698cc3052f93b4a0dfafa8cec1a57b0bace2d2b1d6d not found: ID does not exist"
Nov 03 00:24:44.332422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.332352    4550 scope.go:122] "RemoveContainer" containerID="561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"
Nov 03 00:24:44.332708 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:24:44.332685    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b\": container with ID starting with 561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b not found: ID does not exist" containerID="561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"
Nov 03 00:24:44.332819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.332795    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b"} err="failed to get container status \"561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b\": rpc error: code = NotFound desc = could not find container \"561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b\": container with ID starting with 561d52587b39591017504a65eb0d8626ce492e0aa57ba2ca74e5e0548f3a625b not found: ID does not exist"
Nov 03 00:24:44.354834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.354805    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"]
Nov 03 00:24:44.359840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.359811    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-714512-openshift-default-8fb84d547-nv6vg"]
Nov 03 00:24:44.435486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:44.435452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:45.809635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:45.809602    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" path="/var/lib/kubelet/pods/ddcd8a18-6b88-449a-a32e-9fd43b678269/volumes"
Nov 03 00:24:46.309677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:46.309633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:48.011964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:48.011929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:48.361154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:48.361058    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"]
Nov 03 00:24:48.451484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:48.451417    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:49.710628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:49.710572    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:51.401839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:51.401804    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:52.981897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:52.981861    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:54.568759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:54.568724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:54.725257 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:24:54.725221    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:24:56.226209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:56.226168    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:58.158024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:58.157984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:59.096092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:59.096057    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:59.653814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:59.653769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:24:59.825055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:24:59.825017    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:00.445784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:00.445740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:01.029655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:01.029615    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"]
Nov 03 00:25:01.042773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:01.042744    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b106dd79-fn7hv"]
Nov 03 00:25:01.748243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:01.748206    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:01.810688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:01.810646    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811dd584-4e82-4882-a8b4-f2536e2e3ef3" path="/var/lib/kubelet/pods/811dd584-4e82-4882-a8b4-f2536e2e3ef3/volumes"
Nov 03 00:25:02.483847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:02.483812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:02.792211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:02.792120    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:03.405215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.405168    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerName="main" containerID="cri-o://2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5" gracePeriod=45
Nov 03 00:25:03.695276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.695243    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:25:03.787661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.787627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:03.833384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.833341    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm\") pod \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") "
Nov 03 00:25:03.833633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.833483    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs\") pod \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") "
Nov 03 00:25:03.833633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.833570    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home\") pod \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") "
Nov 03 00:25:03.833745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.833720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr\" (UniqueName: \"kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr\") pod \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") "
Nov 03 00:25:03.833846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.833815    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir\") pod \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") "
Nov 03 00:25:03.833994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.833888    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache\") pod \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\" (UID: \"8255effd-ebb8-4eee-9730-c41ae7c18ee7\") "
Nov 03 00:25:03.835121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.834855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home" pod "8255effd-ebb8-4eee-9730-c41ae7c18ee7" (UID: "8255effd-ebb8-4eee-9730-c41ae7c18ee7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:03.835288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.835255    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir" pod "8255effd-ebb8-4eee-9730-c41ae7c18ee7" (UID: "8255effd-ebb8-4eee-9730-c41ae7c18ee7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:03.837860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.835628    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache" pod "8255effd-ebb8-4eee-9730-c41ae7c18ee7" (UID: "8255effd-ebb8-4eee-9730-c41ae7c18ee7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:03.837860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.837737    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm" pod "8255effd-ebb8-4eee-9730-c41ae7c18ee7" (UID: "8255effd-ebb8-4eee-9730-c41ae7c18ee7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:25:03.849908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.849880    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs" pod "8255effd-ebb8-4eee-9730-c41ae7c18ee7" (UID: "8255effd-ebb8-4eee-9730-c41ae7c18ee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:03.854990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.854952    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr" pod "8255effd-ebb8-4eee-9730-c41ae7c18ee7" (UID: "8255effd-ebb8-4eee-9730-c41ae7c18ee7"). InnerVolumeSpecName "kube-api-access-vhfvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:03.908349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.908311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:03.936480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.936372    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:25:03.936480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.936405    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:25:03.936480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.936422    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhfvr\" (UniqueName: \"kubernetes.io/projected/8255effd-ebb8-4eee-9730-c41ae7c18ee7-kube-api-access-vhfvr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:25:03.936480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.936458    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:25:03.936480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.936479    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:25:03.936778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:03.936492    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8255effd-ebb8-4eee-9730-c41ae7c18ee7-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:25:04.474198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.474160    4550 generic.go:356] "Generic (PLEG): container finished" podID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerID="2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5" exitCode=0
Nov 03 00:25:04.474420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.474210    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"
Nov 03 00:25:04.474420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.474249    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" event={"ID":"8255effd-ebb8-4eee-9730-c41ae7c18ee7","Type":"ContainerDied","Data":"2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5"}
Nov 03 00:25:04.474420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.474319    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb" event={"ID":"8255effd-ebb8-4eee-9730-c41ae7c18ee7","Type":"ContainerDied","Data":"6c7ea0f1fb74aaba6be48f573787e41d042dacfb56b6fa934e6b5cfd9037232c"}
Nov 03 00:25:04.474420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.474348    4550 scope.go:122] "RemoveContainer" containerID="2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5"
Nov 03 00:25:04.485636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.485615    4550 scope.go:122] "RemoveContainer" containerID="2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5"
Nov 03 00:25:04.485943 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:25:04.485925    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5\": container with ID starting with 2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5 not found: ID does not exist" containerID="2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5"
Nov 03 00:25:04.486014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.485954    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5"} err="failed to get container status \"2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5\": rpc error: code = NotFound desc = could not find container \"2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5\": container with ID starting with 2835f2115e512f2b3d9e02fb30ea8a711ca444374bd8ba9db74a6f66aaef3db5 not found: ID does not exist"
Nov 03 00:25:04.532787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.532752    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"]
Nov 03 00:25:04.545641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.545611    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9f76b708/e2e-lifecycle-model-9f76b708-kserve-db7898745-7x5gb"]
Nov 03 00:25:04.568328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:04.568267    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:05.030323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:05.030275    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"]
Nov 03 00:25:05.043311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:05.043277    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fb37e5-2qb82"]
Nov 03 00:25:05.494937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:05.494907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:05.602260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:05.602220    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:05.910711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:05.908516    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030893b9-f5dd-49e3-9320-849985c23f2a" path="/var/lib/kubelet/pods/030893b9-f5dd-49e3-9320-849985c23f2a/volumes"
Nov 03 00:25:05.910711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:05.909511    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" path="/var/lib/kubelet/pods/8255effd-ebb8-4eee-9730-c41ae7c18ee7/volumes"
Nov 03 00:25:06.500108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:06.500065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:07.035845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.035786    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"]
Nov 03 00:25:07.036619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036601    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy"
Nov 03 00:25:07.036699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036622    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy"
Nov 03 00:25:07.036699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036635    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy"
Nov 03 00:25:07.036699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036643    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy"
Nov 03 00:25:07.036699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036687    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerName="main"
Nov 03 00:25:07.036699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036695    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerName="main"
Nov 03 00:25:07.036845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036810    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8255effd-ebb8-4eee-9730-c41ae7c18ee7" containerName="main"
Nov 03 00:25:07.036845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036825    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy"
Nov 03 00:25:07.036845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.036833    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcd8a18-6b88-449a-a32e-9fd43b678269" containerName="istio-proxy"
Nov 03 00:25:07.040872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.040849    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.060810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.060782    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4aa4e9-openshift-default-dockercfg-fkwlw\""
Nov 03 00:25:07.067174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.067143    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"]
Nov 03 00:25:07.075410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075378    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075466    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075504    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075577    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075626    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075654    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075687    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.075872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.075711    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvkjz\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177380    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177537    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177582    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177621    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177668    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177712    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177763    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177802    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.177941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.177828    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvkjz\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.178081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.178056    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.178142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.178111    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.178348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.178312    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.178648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.178625    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.178841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.178817    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.180468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.180124    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.180468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.180351    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.205263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.205226    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvkjz\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.205390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.205244    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.357212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.357120    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:07.681501 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:25:07.681426    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1015f95c_4f7a_408e_82e1_6859fdf7c349.slice/crio-2e3f0c5d2eff94efc4e2bd34fa4092841f95b59450c1e8fde662dadc4405d1c8 WatchSource:0}: Error finding container 2e3f0c5d2eff94efc4e2bd34fa4092841f95b59450c1e8fde662dadc4405d1c8: Status 404 returned error can't find the container with id 2e3f0c5d2eff94efc4e2bd34fa4092841f95b59450c1e8fde662dadc4405d1c8
Nov 03 00:25:07.681912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.681834    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"]
Nov 03 00:25:07.694706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.694660    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:07.694846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.694756    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:07.694846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:07.694798    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:08.084805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:08.084382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:08.188648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:08.188611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:08.356279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:08.356157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:08.521166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:08.521132    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" event={"ID":"1015f95c-4f7a-408e-82e1-6859fdf7c349","Type":"ContainerStarted","Data":"07d29a73e379bfa8a4893c71e0ff4b0b2af85c10f97f8b7ea56fbda9f61aecdf"}
Nov 03 00:25:08.521371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:08.521185    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" event={"ID":"1015f95c-4f7a-408e-82e1-6859fdf7c349","Type":"ContainerStarted","Data":"2e3f0c5d2eff94efc4e2bd34fa4092841f95b59450c1e8fde662dadc4405d1c8"}
Nov 03 00:25:08.610087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:08.609983    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podStartSLOduration=1.609953304 podStartE2EDuration="1.609953304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:08.59352969 +0000 UTC m=+1190.915066488" watchObservedRunningTime="2026-11-03 00:25:08.609953304 +0000 UTC m=+1190.931490104"
Nov 03 00:25:09.097350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:09.097318    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:09.364516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:09.361790    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:09.369459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:09.366557    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 00:25:09.369459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:09.366620    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 00:25:09.633930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:09.633843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:10.353825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:10.353790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:10.376517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:10.368867    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 00:25:10.376517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:10.368935    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 00:25:10.495909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:10.495819    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:11.001787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:11.001731    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:11.349465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:11.349372    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:11.368386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:11.360550    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 00:25:11.368386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:11.360615    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 00:25:12.360455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:12.359850    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 00:25:12.360455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:12.359932    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 00:25:12.651634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:12.651599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:13.364679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:13.364645    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Nov 03 00:25:13.365018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:13.364972    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Nov 03 00:25:14.512456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:14.511749    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:14.516454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:14.513228    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:14.533455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:14.528960    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:25:15.019795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.019037    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:15.253410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.253366    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"]
Nov 03 00:25:15.297518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.279115    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f73e534c-lkdkd"]
Nov 03 00:25:15.308533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.307681    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:15.517514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.517465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:15.693928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.693886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:15.809033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:15.808989    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4eb8a5b-fe48-4074-9280-305ee799afea" path="/var/lib/kubelet/pods/e4eb8a5b-fe48-4074-9280-305ee799afea/volumes"
Nov 03 00:25:16.181526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:16.181496    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:17.152109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:17.152070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:17.264317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:17.264276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:18.112724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:18.112693    4550 scope.go:122] "RemoveContainer" containerID="5438459c0a8da21c1eb8868174d37f8d7d23ea7053aa5aed1b930e04e1912e76"
Nov 03 00:25:18.133624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:18.133600    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:19.778954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:19.778691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:19.909029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:19.908980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:20.075191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:20.075084    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:21.739793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:21.739763    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:23.418668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:23.418627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:24.791384 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:25:24.791341    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:25:24.985940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:24.985893    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:25.236587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:25.236549    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:26.120795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:26.120750    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:26.216129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:26.216061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:26.342191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:26.342149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:26.661252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:26.661217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:27.155404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:27.155370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:27.255065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:27.255028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:27.365347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:27.365305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:27.978498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:27.978424    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:29.665114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:29.665023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:29.765004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:29.764946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:30.038879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:30.038782    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"]
Nov 03 00:25:30.051116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:30.051083    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82ee2558-xnhrt"]
Nov 03 00:25:30.354161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:30.354063    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:30.810018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:30.809965    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:30.932453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:30.932378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:31.049747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:31.049706    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:31.616273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:31.616235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:31.810386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:31.810353    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9" path="/var/lib/kubelet/pods/7405e1a3-a1a1-46b8-b673-7eb3f0cde9c9/volumes"
Nov 03 00:25:31.848026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:31.847987    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:32.624893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:32.624859    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:32.726889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:32.726852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:33.076380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:33.076347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:33.994876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:33.994837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:34.113609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:34.113564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:34.212119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:34.212086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:36.098621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:36.097716    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:37.922622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:37.922575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:38.035720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:38.035682    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:38.297021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:38.296922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:40.145974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:40.145942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:41.723887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:41.723852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:42.854924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:42.854887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:43.670968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:43.670935    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:44.334349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:44.334307    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:44.852342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:44.852307    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:45.004027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:45.003993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:45.032298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:45.032265    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"]
Nov 03 00:25:45.045356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:45.045320    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de5e6880-rl4zg"]
Nov 03 00:25:45.625114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:45.625080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:45.735217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:45.735185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:45.858646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:45.858596    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a94670-7515-43f1-8218-897123cc4f03" path="/var/lib/kubelet/pods/61a94670-7515-43f1-8218-897123cc4f03/volumes"
Nov 03 00:25:47.143170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:47.143131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:49.005722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:49.005680    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:49.111220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:49.111171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:49.215303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:49.215263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:49.425723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:49.425684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:49.528378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:49.528336    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:49.889174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:49.889081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:50.255744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:50.255713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:51.048138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.048108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:51.272401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.272361    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:51.524856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.524815    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:51.607888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.607847    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"]
Nov 03 00:25:51.618474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.618422    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.624058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.624026    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"]
Nov 03 00:25:51.642217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.642071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:51.668159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.668113    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-27490b-openshift-default-dockercfg-njcpl\""
Nov 03 00:25:51.710810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.710768    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.710998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.710863    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.710998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.710932    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.710998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.710954    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.710998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.710979    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpm6j\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.711162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.711144    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.711198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.711172    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.711312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.711291    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.711376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.711330    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.749832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.749794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:51.812860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.812734    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.812860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.812797    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.812861    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.812892    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.812932    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpm6j\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.813054    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.813080    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.813147    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.813176    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.813806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.813702    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.836924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.836887    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.837212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.837147    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.839645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.838152    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.839645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.838502    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.859504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.855533    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.863451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.860543    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.879879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.879846    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpm6j\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.920086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.920048    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token\") pod \"e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") " pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:51.951963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:51.951928    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:52.229495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.224910    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"]
Nov 03 00:25:52.255404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.255368    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"]
Nov 03 00:25:52.255634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.255496    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346205    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346282    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346343    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346414    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4x4w\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346481    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346516    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346583    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346637    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.346936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.346720    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.353052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.353011    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"]
Nov 03 00:25:52.354177 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:25:52.354144    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d23d244_426e_4e20_a64e_ab3ff625ff00.slice/crio-d44d61b12f7fee7fca0d15e6a390fe78a10d5efd233abffa2f0382aa57ef0374 WatchSource:0}: Error finding container d44d61b12f7fee7fca0d15e6a390fe78a10d5efd233abffa2f0382aa57ef0374: Status 404 returned error can't find the container with id d44d61b12f7fee7fca0d15e6a390fe78a10d5efd233abffa2f0382aa57ef0374
Nov 03 00:25:52.356900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.356864    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.357015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.356934    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.357015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.356963    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.447816 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.447784    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.447983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.447882    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.447983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.447927    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.447999    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448074    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4x4w\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448105    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448135    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448200    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448255    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448256    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448311    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448578    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448711    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.448885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.448804    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.450538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.450515    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.451164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.451141    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.473483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.468938    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.473483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.469531    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4x4w\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.580451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.579640    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:52.921753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.921721    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"]
Nov 03 00:25:52.923635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.923595    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.923754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.923672    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:52.923754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:52.923716    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:25:53.260488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.260345    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" event={"ID":"bf175420-7940-47a0-b46e-7419de2d9735","Type":"ContainerStarted","Data":"0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec"}
Nov 03 00:25:53.260488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.260415    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" event={"ID":"bf175420-7940-47a0-b46e-7419de2d9735","Type":"ContainerStarted","Data":"31e8486f0f2ed4ca2ba89b355e23dbca73e86217da09eeb5f544b55f0a298ae8"}
Nov 03 00:25:53.262838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.262782    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" event={"ID":"9d23d244-426e-4e20-a64e-ab3ff625ff00","Type":"ContainerStarted","Data":"889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"}
Nov 03 00:25:53.262838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.262837    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" event={"ID":"9d23d244-426e-4e20-a64e-ab3ff625ff00","Type":"ContainerStarted","Data":"d44d61b12f7fee7fca0d15e6a390fe78a10d5efd233abffa2f0382aa57ef0374"}
Nov 03 00:25:53.305257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.305199    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podStartSLOduration=1.305180567 podStartE2EDuration="1.305180567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:53.305169876 +0000 UTC m=+1235.626706679" watchObservedRunningTime="2026-11-03 00:25:53.305180567 +0000 UTC m=+1235.626717372"
Nov 03 00:25:53.351339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.351274    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podStartSLOduration=2.351255045 podStartE2EDuration="2.351255045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:53.347476739 +0000 UTC m=+1235.669013541" watchObservedRunningTime="2026-11-03 00:25:53.351255045 +0000 UTC m=+1235.672791846"
Nov 03 00:25:53.467209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.467171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:53.580759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.580667    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:53.952317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.952280    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:53.974465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:53.973776    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:54.293053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:54.292970    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:54.294181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:54.294154    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:25:54.588519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:54.587576    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 00:25:54.588519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:54.587642    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 00:25:54.726171 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:25:54.726140    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:25:55.436706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:55.436672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:55.582467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:55.581892    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 00:25:55.582467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:55.581957    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 00:25:55.852970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:55.852894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:55.965969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:55.965933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:56.032453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:56.032407    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"]
Nov 03 00:25:56.045423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:56.045391    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd182881-pjkvh"]
Nov 03 00:25:56.581131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:56.581082    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 00:25:56.581353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:56.581154    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 00:25:57.317458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:57.317006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:57.594509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:57.594192    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 00:25:57.594509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:57.594262    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 00:25:57.822803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:57.820076    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba679af-a218-4a8f-b1d6-965f96f93b4a" path="/var/lib/kubelet/pods/dba679af-a218-4a8f-b1d6-965f96f93b4a/volumes"
Nov 03 00:25:57.826059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:57.826022    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:57.952934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:57.952898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:58.590904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:58.590867    4550 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Nov 03 00:25:58.591563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:58.591525    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Nov 03 00:25:58.945182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:58.945148    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:25:59.944209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:59.944173    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:59.944752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:59.944474    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:25:59.945275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:25:59.945257    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:26:00.051817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:00.051769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:00.147769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:00.147737    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:00.834921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:00.834871    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:00.934235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:00.934185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:01.035528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.035494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:01.331253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.331193    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"]
Nov 03 00:26:01.332585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.332413    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy" containerID="cri-o://07d29a73e379bfa8a4893c71e0ff4b0b2af85c10f97f8b7ea56fbda9f61aecdf" gracePeriod=30
Nov 03 00:26:01.356376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.356340    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"]
Nov 03 00:26:01.393722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.393679    4550 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" secret="" err="secret \"e2e-api-b-4aa4e9-openshift-default-dockercfg-fkwlw\" not found"
Nov 03 00:26:01.395826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.395795    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:01.954368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:01.954327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:02.400986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:02.400953    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy" containerID="cri-o://0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec" gracePeriod=30
Nov 03 00:26:02.653706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:02.653615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:03.158191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:03.158146    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:03.256459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:03.255446    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:04.086360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:04.086312    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:06.029331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.029288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:06.506813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.506772    4550 generic.go:356] "Generic (PLEG): container finished" podID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerID="07d29a73e379bfa8a4893c71e0ff4b0b2af85c10f97f8b7ea56fbda9f61aecdf" exitCode=0
Nov 03 00:26:06.506995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.506825    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" event={"ID":"1015f95c-4f7a-408e-82e1-6859fdf7c349","Type":"ContainerDied","Data":"07d29a73e379bfa8a4893c71e0ff4b0b2af85c10f97f8b7ea56fbda9f61aecdf"}
Nov 03 00:26:06.625843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.625815    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:26:06.708313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708273    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708341    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708416    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708461    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708518    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708601    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708659    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708713    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708765    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket\") pod \"1015f95c-4f7a-408e-82e1-6859fdf7c349\" (UID: \"1015f95c-4f7a-408e-82e1-6859fdf7c349\") "
Nov 03 00:26:06.708959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708812    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.708959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.708891    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.709351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709116    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.709351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709173    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.709482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709349    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:06.709482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709377    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.709482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709397    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.709482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709411    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.709482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.709426    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.711276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.711248    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:06.711396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.711358    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:06.711699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.711669    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "kube-api-access-bvkjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:06.711780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.711762    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo" pod "1015f95c-4f7a-408e-82e1-6859fdf7c349" (UID: "1015f95c-4f7a-408e-82e1-6859fdf7c349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:06.811489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.811460    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.811489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.811490    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvkjz\" (UniqueName: \"kubernetes.io/projected/1015f95c-4f7a-408e-82e1-6859fdf7c349-kube-api-access-bvkjz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.811692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.811514    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.811692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.811527    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1015f95c-4f7a-408e-82e1-6859fdf7c349-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:06.811692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:06.811539    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1015f95c-4f7a-408e-82e1-6859fdf7c349-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.516563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.516528    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b" event={"ID":"1015f95c-4f7a-408e-82e1-6859fdf7c349","Type":"ContainerDied","Data":"2e3f0c5d2eff94efc4e2bd34fa4092841f95b59450c1e8fde662dadc4405d1c8"}
Nov 03 00:26:07.516563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.516565    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"
Nov 03 00:26:07.517075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.516593    4550 scope.go:122] "RemoveContainer" containerID="07d29a73e379bfa8a4893c71e0ff4b0b2af85c10f97f8b7ea56fbda9f61aecdf"
Nov 03 00:26:07.595333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.595291    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"]
Nov 03 00:26:07.603757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.603726    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-n5t4b"]
Nov 03 00:26:07.697022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.696996    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:26:07.726514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726471    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726586    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726688    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726730    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726758    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726790    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726842    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726904    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.726951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.726934    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token\") pod \"bf175420-7940-47a0-b46e-7419de2d9735\" (UID: \"bf175420-7940-47a0-b46e-7419de2d9735\") "
Nov 03 00:26:07.727106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727051    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.727175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727100    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.727229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727164    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.727229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727177    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.727229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727172    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:07.727717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727689    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf175420-7940-47a0-b46e-7419de2d9735-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.727717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727719    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.727907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727733    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.727907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727746    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.727907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.727761    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.729072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.729045    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:07.729169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.729111    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.729169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.729122    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.729169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.729134    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w" pod "bf175420-7940-47a0-b46e-7419de2d9735" (UID: "bf175420-7940-47a0-b46e-7419de2d9735"). InnerVolumeSpecName "kube-api-access-c4x4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.781260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.781206    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:07.812003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.811970    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" path="/var/lib/kubelet/pods/1015f95c-4f7a-408e-82e1-6859fdf7c349/volumes"
Nov 03 00:26:07.828746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.828721    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf175420-7940-47a0-b46e-7419de2d9735-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.828746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.828744    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.828932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.828753    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf175420-7940-47a0-b46e-7419de2d9735-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.828932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.828763    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4x4w\" (UniqueName: \"kubernetes.io/projected/bf175420-7940-47a0-b46e-7419de2d9735-kube-api-access-c4x4w\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.889643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:07.889596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:08.526499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.526461    4550 generic.go:356] "Generic (PLEG): container finished" podID="bf175420-7940-47a0-b46e-7419de2d9735" containerID="0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec" exitCode=0
Nov 03 00:26:08.526917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.526508    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"
Nov 03 00:26:08.526917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.526541    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" event={"ID":"bf175420-7940-47a0-b46e-7419de2d9735","Type":"ContainerDied","Data":"0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec"}
Nov 03 00:26:08.526917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.526595    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp" event={"ID":"bf175420-7940-47a0-b46e-7419de2d9735","Type":"ContainerDied","Data":"31e8486f0f2ed4ca2ba89b355e23dbca73e86217da09eeb5f544b55f0a298ae8"}
Nov 03 00:26:08.526917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.526618    4550 scope.go:122] "RemoveContainer" containerID="0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec"
Nov 03 00:26:08.536825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.536799    4550 scope.go:122] "RemoveContainer" containerID="0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec"
Nov 03 00:26:08.537104 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:26:08.537081    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec\": container with ID starting with 0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec not found: ID does not exist" containerID="0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec"
Nov 03 00:26:08.537197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.537117    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec"} err="failed to get container status \"0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec\": rpc error: code = NotFound desc = could not find container \"0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec\": container with ID starting with 0e74341a13cfff5f78aabf84c359660e5a64f53cd269fa323919427910f27dec not found: ID does not exist"
Nov 03 00:26:08.598711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.598678    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"]
Nov 03 00:26:08.611669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:08.611635    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-767wp"]
Nov 03 00:26:09.035051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.035017    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"]
Nov 03 00:26:09.039473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.039444    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-853dc0c3-p66rl"]
Nov 03 00:26:09.485195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.485102    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:09.586898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.586858    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:09.753158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.753072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:09.905085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.903873    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf175420-7940-47a0-b46e-7419de2d9735" path="/var/lib/kubelet/pods/bf175420-7940-47a0-b46e-7419de2d9735/volumes"
Nov 03 00:26:09.905085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.904618    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff04cf16-0d69-4721-9f9d-c4509735df70" path="/var/lib/kubelet/pods/ff04cf16-0d69-4721-9f9d-c4509735df70/volumes"
Nov 03 00:26:09.993392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:09.993361    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:10.104897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:10.104796    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:10.189987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:10.189952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:12.024813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:12.024770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:13.987881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:13.987835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:16.135568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:16.135525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:16.645222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:16.645179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:17.328040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:17.327996    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:17.822386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:17.822328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:18.009798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.009606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:18.110102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.109995    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:18.164585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.161950    4550 scope.go:122] "RemoveContainer" containerID="fc0327a3b9891816e2f6957e5cc7882c482102a14676d4807f6031e74178eb99"
Nov 03 00:26:18.197787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.197756    4550 scope.go:122] "RemoveContainer" containerID="8f2e61483876ffbd6ee55b788d52a0e1f29334d42ad7c510d7550da121f95806"
Nov 03 00:26:18.248860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.248822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:18.255921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.255898    4550 scope.go:122] "RemoveContainer" containerID="b8381463874f3ffab044b967e96dca2fd0d2c100da6a5a674a3aaf51eeccb675"
Nov 03 00:26:18.374637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:18.374542    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:20.033534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:20.033495    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"]
Nov 03 00:26:20.048026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:20.047993    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f16327f3-v7bsp"]
Nov 03 00:26:20.191119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:20.191078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:21.590107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:21.590076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:21.769826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:21.769775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:21.808636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:21.808600    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472204c2-e688-4adc-83b6-a23e4d5db63c" path="/var/lib/kubelet/pods/472204c2-e688-4adc-83b6-a23e4d5db63c/volumes"
Nov 03 00:26:21.946587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:21.946553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:22.257832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.257730    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"]
Nov 03 00:26:22.261588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.261558    4550 cpu_manager.go:414] "containerMap: removing container" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy"
Nov 03 00:26:22.261588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.261588    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy"
Nov 03 00:26:22.261588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.261630    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy"
Nov 03 00:26:22.261588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.261639    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy"
Nov 03 00:26:22.262808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.261793    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1015f95c-4f7a-408e-82e1-6859fdf7c349" containerName="istio-proxy"
Nov 03 00:26:22.262808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.261813    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf175420-7940-47a0-b46e-7419de2d9735" containerName="istio-proxy"
Nov 03 00:26:22.294537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.294505    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"]
Nov 03 00:26:22.294831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.294817    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.317750 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.317723    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41e815\""
Nov 03 00:26:22.317944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.317798    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41e815-dockercfg-hgspm\""
Nov 03 00:26:22.350378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.350339    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.350599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.350516    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s789\" (UniqueName: \"kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789\") pod \"payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.451972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.451936    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.452196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.452105    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s789\" (UniqueName: \"kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789\") pod \"payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.452681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.452663    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.475740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.475704    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s789\" (UniqueName: \"kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789\") pod \"payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.621615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.621520    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:22.928859 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:26:22.928822    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf474dc7f_810f_4ebf_9518_6aa5df9e994f.slice/crio-b7823232778316178ce1dcd1dd327156c37457b42e94c4723becbca3714e9e00 WatchSource:0}: Error finding container b7823232778316178ce1dcd1dd327156c37457b42e94c4723becbca3714e9e00: Status 404 returned error can't find the container with id b7823232778316178ce1dcd1dd327156c37457b42e94c4723becbca3714e9e00
Nov 03 00:26:22.930339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:22.930309    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"]
Nov 03 00:26:23.077195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:23.077160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:23.409131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:23.409096    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:23.508951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:23.508914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:23.756719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:23.734406    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc" event={"ID":"f474dc7f-810f-4ebf-9518-6aa5df9e994f","Type":"ContainerStarted","Data":"b7823232778316178ce1dcd1dd327156c37457b42e94c4723becbca3714e9e00"}
Nov 03 00:26:24.725142 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:26:24.725110    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:26:24.744131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:24.744091    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc" event={"ID":"f474dc7f-810f-4ebf-9518-6aa5df9e994f","Type":"ContainerStarted","Data":"091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20"}
Nov 03 00:26:24.744333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:24.744226    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:24.787386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:24.787330    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc" podStartSLOduration=1.06694332 podStartE2EDuration="2.787314187s" totalImagesPullingTime="1.720370867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:24.78506548 +0000 UTC m=+1267.106602281" watchObservedRunningTime="2026-11-03 00:26:24.787314187 +0000 UTC m=+1267.108850987"
Nov 03 00:26:25.020544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:25.020320    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:26.848206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:26.848166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:28.538909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:28.538867    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:30.154076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:30.154035    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:30.251075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:30.251041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:30.393347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:30.393315    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:32.128944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:32.128898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:33.931711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:33.931649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:35.753319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:35.753287    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:26:36.010142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:36.010045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:37.781085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:37.781004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:37.895202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:37.895168    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:38.303782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.303750    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"]
Nov 03 00:26:38.314012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.313967    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.336199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.336154    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"]
Nov 03 00:26:38.345524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.344018    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dda6df-openshift-default-dockercfg-p28rh\""
Nov 03 00:26:38.346044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346014    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346171    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346300    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346407    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnx4\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346469    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346524    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346561    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346645    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.346756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.346720    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.393358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.393300    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:38.447632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.447601    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.447803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.447661    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.447803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.447766    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.447940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.447849    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnx4\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.447940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.447892    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448200    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448248    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448336    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448407    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448503    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448210    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.448807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.448766    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.449208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.449183    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.449632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.449608    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.450568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.450550    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.450951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.450928    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.480501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.479706    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.492453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.482545    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnx4\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4\") pod \"e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") " pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:38.507456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.501596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:38.644507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:38.643879    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:39.145508 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:26:39.145094    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b1cea3_144f_40c0_91c9_12f1d3ff93ec.slice/crio-1383fd027ec12657acc8a41c8b5c62bb116cdcb2116ff568eaa1a899cf771c34 WatchSource:0}: Error finding container 1383fd027ec12657acc8a41c8b5c62bb116cdcb2116ff568eaa1a899cf771c34: Status 404 returned error can't find the container with id 1383fd027ec12657acc8a41c8b5c62bb116cdcb2116ff568eaa1a899cf771c34
Nov 03 00:26:39.146890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.146869    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"]
Nov 03 00:26:39.147757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.147725    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:39.147815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.147796    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:39.147853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.147834    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:39.736798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.736761    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:39.921405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.921353    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" event={"ID":"55b1cea3-144f-40c0-91c9-12f1d3ff93ec","Type":"ContainerStarted","Data":"79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"}
Nov 03 00:26:39.921611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.921420    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" event={"ID":"55b1cea3-144f-40c0-91c9-12f1d3ff93ec","Type":"ContainerStarted","Data":"1383fd027ec12657acc8a41c8b5c62bb116cdcb2116ff568eaa1a899cf771c34"}
Nov 03 00:26:39.995514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:39.994728    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podStartSLOduration=1.994709248 podStartE2EDuration="1.994709248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:39.984803031 +0000 UTC m=+1282.306339832" watchObservedRunningTime="2026-11-03 00:26:39.994709248 +0000 UTC m=+1282.316246050"
Nov 03 00:26:40.187461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:40.185282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:40.645242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:40.645207    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:40.652072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:40.652048    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:40.928792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:40.928761    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:40.929774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:40.929756    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:26:41.294095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:41.293997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:41.512023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:41.511987    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:41.660792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:41.660701    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:42.356790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:42.356758    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:42.460077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:42.460041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:43.792729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:43.792662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:44.957483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:44.957447    4550 generic.go:356] "Generic (PLEG): container finished" podID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerID="889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9" exitCode=0
Nov 03 00:26:44.957483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:44.957462    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" event={"ID":"9d23d244-426e-4e20-a64e-ab3ff625ff00","Type":"ContainerDied","Data":"889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"}
Nov 03 00:26:44.958065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:44.958014    4550 scope.go:122] "RemoveContainer" containerID="889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"
Nov 03 00:26:44.959915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:44.959879    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:44.960026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:44.959955    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:44.960026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:44.959994    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:45.175905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:45.174717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:45.286254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:45.286160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:45.967503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:45.967461    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" event={"ID":"9d23d244-426e-4e20-a64e-ab3ff625ff00","Type":"ContainerStarted","Data":"62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c"}
Nov 03 00:26:46.436653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:46.436616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:46.537246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:46.537200    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:46.683455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:46.682936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:46.965473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:46.964830    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:26:46.965473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:46.964966    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 00:26:46.965473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:46.965024    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 00:26:47.333900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.333799    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:47.607164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.607073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:47.728694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.728656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:47.808464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.800121    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"]
Nov 03 00:26:47.925198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.849658    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:47.928571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.928542    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:47.958966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.958919    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 00:26:47.996465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.990567    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 00:26:47.996465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:47.971956    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1d29dd-openshift-default-dockercfg-vggqd\""
Nov 03 00:26:48.113872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.113833    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.114264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.114239    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.114583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.114528    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.115000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.114981    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.115175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.115158    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.115306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.115293    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.115444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.115415    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.115566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.115550    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.230727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.219735    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85k6f\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.231388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.199380    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"]
Nov 03 00:26:48.234179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.172928    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:48.333311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.333276    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.333741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.333720    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.333947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.333928    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.334467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.334448    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.334653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.334625    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.335518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.335486    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.335637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.335517    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.335637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.335611    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k6f\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.335763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.335672    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.335763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.335726    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.335922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.335792    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.336100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.336074    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.336961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.336940    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.337389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.337361    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.353187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.353145    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.356297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.356257    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.401526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.394406    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.435380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.435345    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k6f\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f\") pod \"e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") " pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.457720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.457686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:48.592604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.592510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:48.726548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.726508    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:48.974905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.968155    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 00:26:48.974905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:48.968234    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 00:26:49.050476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.047607    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:49.222629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.213205    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"]
Nov 03 00:26:49.235016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.234969    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:49.235156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.235066    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:49.235156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.235108    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:26:49.953137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.953107    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 00:26:49.953551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:49.953507    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 00:26:50.222556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.222457    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" event={"ID":"15f56704-a672-4db5-acd4-3708cef007c8","Type":"ContainerStarted","Data":"3e9b5987ba525891adc2af7b67206d73e4fc0770222243dfb02b107511c53c4a"}
Nov 03 00:26:50.222556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.222529    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" event={"ID":"15f56704-a672-4db5-acd4-3708cef007c8","Type":"ContainerStarted","Data":"465a2d61861adef603bdd47831889ecfd66106d3d53127857a8ac88ba2573e7d"}
Nov 03 00:26:50.293163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.293103    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podStartSLOduration=3.293088254 podStartE2EDuration="3.293088254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:50.278963033 +0000 UTC m=+1292.600499834" watchObservedRunningTime="2026-11-03 00:26:50.293088254 +0000 UTC m=+1292.614625095"
Nov 03 00:26:50.510171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.510079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:50.739489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.739460    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:50.787951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.787856    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:50.952718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.952680    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 00:26:50.953019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:50.952987    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 00:26:51.255510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:51.255331    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:51.261990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:51.261662    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:26:51.482112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:51.482051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:51.953045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:51.953010    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Nov 03 00:26:51.953235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:51.953065    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Nov 03 00:26:52.758264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:52.758224    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:52.866897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:52.866853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:52.958114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:52.958078    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:26:52.958662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:52.958642    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:26:52.959377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:52.959360    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:26:53.746745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:53.746704    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:54.032922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:54.032843    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"]
Nov 03 00:26:54.045885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:54.045855    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c433e695-b7rrj"]
Nov 03 00:26:54.654908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:54.649599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:54.725692 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:26:54.725651    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:26:54.745538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:54.745505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:55.095525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:55.092153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:55.190928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:55.190871    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:55.807713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:55.807678    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4202c346-e09f-4152-bfae-9b609acba24b" path="/var/lib/kubelet/pods/4202c346-e09f-4152-bfae-9b609acba24b/volumes"
Nov 03 00:26:55.884851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:55.884818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:56.121772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:56.121676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:56.297936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:56.297894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:57.221390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:57.221355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:58.039423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:58.039389    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"]
Nov 03 00:26:58.054480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:58.052566    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0aee6b3-v4zj9"]
Nov 03 00:26:58.170795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:58.170762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:58.828118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:58.828084    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:58.926489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:58.926419    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:59.026497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:59.026462    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:26:59.810329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:26:59.810245    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8567e234-60b5-4364-a0b7-68b83abd51ec" path="/var/lib/kubelet/pods/8567e234-60b5-4364-a0b7-68b83abd51ec/volumes"
Nov 03 00:27:00.336727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:00.336694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:02.296550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:02.296515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:02.737653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:02.737616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:02.838619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:02.838580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:03.662154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:03.662118    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:04.252631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:04.252595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:06.240920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:06.240879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:07.980800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:07.980763    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:08.033865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:08.033832    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"]
Nov 03 00:27:08.047834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:08.047798    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0397f2d-hrchl"]
Nov 03 00:27:08.080232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:08.080185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:08.183811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:08.183766    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:09.299987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:09.299951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:09.809684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:09.809654    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e5c23d-389c-44b3-83b3-3264e8d0ffa1" path="/var/lib/kubelet/pods/a8e5c23d-389c-44b3-83b3-3264e8d0ffa1/volumes"
Nov 03 00:27:09.999889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:09.999847    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:10.097456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:10.097350    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:11.824487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:11.824456    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:14.138392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:14.138359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:15.883483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:15.883445    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:17.728118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:17.728080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:17.821057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:17.821018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:17.921514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:17.921476    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:18.022722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:18.022137    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:18.358669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:18.358582    4550 scope.go:122] "RemoveContainer" containerID="3d77ce0ce1acfe987e80b89e5b7c6fea0ebd441e8f5ac879029d53e8cbef227b"
Nov 03 00:27:18.370541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:18.370520    4550 scope.go:122] "RemoveContainer" containerID="97cf87c4ec35928639cbe9c30a0b560434f3e22572f12558d7bf4daf09e93270"
Nov 03 00:27:18.381161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:18.381114    4550 scope.go:122] "RemoveContainer" containerID="477c81e81df086be3a57de9dc2389a5c5fa5c0d29c1947dd8d0f3e7d27a50b60"
Nov 03 00:27:18.391039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:18.391016    4550 scope.go:122] "RemoveContainer" containerID="a34ca47b35d5b039a4a8d8af263cd040c7ed718b91b1ba4e84c06fda178ffd19"
Nov 03 00:27:18.401218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:18.401198    4550 scope.go:122] "RemoveContainer" containerID="7e32e2d424c2ea84c4c924565a2266471b48639f0a0d9289512db056c1c49220"
Nov 03 00:27:19.191018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:19.190978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:19.758055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:19.758020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:21.621033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:21.620997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:22.028568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:22.028531    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"]
Nov 03 00:27:22.041291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:22.041261    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e33b8b94-7frzw"]
Nov 03 00:27:23.382759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:23.382707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:23.809105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:23.809070    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1c9071-76a8-465f-9ca2-b9b3630128b5" path="/var/lib/kubelet/pods/6c1c9071-76a8-465f-9ca2-b9b3630128b5/volumes"
Nov 03 00:27:24.687950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:24.687918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:24.724980 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:27:24.724944    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:27:24.847659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:24.847614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:25.575270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:25.575231    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:25.678267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:25.678235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:26.033369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:26.033316    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:26.291232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:26.291134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:27.826917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:27.826872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:28.279454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:28.279315    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:29.302237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:29.302198    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:30.913372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:30.913333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:31.904910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:31.904879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:32.004323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:32.004288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:32.723094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:32.723053    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:32.923881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:32.923849    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:33.122347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:33.122227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:33.919797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:33.919762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:34.043749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:34.043710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:34.666128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:34.666088    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:35.109183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:35.109094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:35.224991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:35.224191    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:37.153505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:37.153463    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:38.452277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:38.452243    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:38.551190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:38.551154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:38.646089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:38.646058    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:39.688259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:39.688105    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:39.798833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:39.798635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:39.897482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:39.897426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:40.487941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:40.487900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:41.184069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:41.184028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:41.652092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:41.651924    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:41.903218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:41.903123    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:42.374513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:42.374218    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:43.514464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:43.512476    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:45.247478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:45.246642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:45.347581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:45.346596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:45.477161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:45.477126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:46.588506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:46.584080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:46.930187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:46.930150    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:47.039185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:47.039149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:47.594534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:47.594444    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:48.764212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:48.764115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:49.460956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:49.460866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:49.917313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:49.917270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:50.508167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:50.493196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:50.676607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:50.676370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:51.165637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.165474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:51.400230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.400193    4550 generic.go:356] "Generic (PLEG): container finished" podID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerID="79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39" exitCode=0
Nov 03 00:27:51.400412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.400229    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" event={"ID":"55b1cea3-144f-40c0-91c9-12f1d3ff93ec","Type":"ContainerDied","Data":"79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"}
Nov 03 00:27:51.400800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.400781    4550 scope.go:122] "RemoveContainer" containerID="79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"
Nov 03 00:27:51.402975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.402938    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:51.403075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.403037    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:51.403120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.403091    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:51.696346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.693086    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": read tcp 10.133.0.2:42688->10.133.0.92:15021: read: connection reset by peer" start-of-body=
Nov 03 00:27:51.696346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.693148    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": read tcp 10.133.0.2:42688->10.133.0.92:15021: read: connection reset by peer"
Nov 03 00:27:51.911038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:51.911003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:52.408027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.407992    4550 generic.go:356] "Generic (PLEG): container finished" podID="15f56704-a672-4db5-acd4-3708cef007c8" containerID="3e9b5987ba525891adc2af7b67206d73e4fc0770222243dfb02b107511c53c4a" exitCode=0
Nov 03 00:27:52.408507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.408067    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" event={"ID":"15f56704-a672-4db5-acd4-3708cef007c8","Type":"ContainerDied","Data":"3e9b5987ba525891adc2af7b67206d73e4fc0770222243dfb02b107511c53c4a"}
Nov 03 00:27:52.408600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.408563    4550 scope.go:122] "RemoveContainer" containerID="3e9b5987ba525891adc2af7b67206d73e4fc0770222243dfb02b107511c53c4a"
Nov 03 00:27:52.410227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.410205    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" event={"ID":"55b1cea3-144f-40c0-91c9-12f1d3ff93ec","Type":"ContainerStarted","Data":"5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695"}
Nov 03 00:27:52.410335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.410248    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:52.410335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.410323    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:52.410529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.410376    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:27:52.645522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.645483    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:27:52.646264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.646232    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 00:27:52.646451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:52.646294    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 00:27:53.420572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.420536    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" event={"ID":"15f56704-a672-4db5-acd4-3708cef007c8","Type":"ContainerStarted","Data":"27eda38831f13dda4d048cd8e0fdc419614caec62f4cba0773aa44a1c0f4a3fa"}
Nov 03 00:27:53.635806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.635765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:53.645745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.645711    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 00:27:53.645910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.645767    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 00:27:53.729516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.728836    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:27:53.729516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.728999    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 00:27:53.729516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:53.729047    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 00:27:54.187460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:54.178366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:54.299064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:54.299032    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:54.645154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:54.645068    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 00:27:54.645154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:54.645136    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 00:27:54.727612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:54.727580    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 00:27:54.727868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:54.727840    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 00:27:54.732811 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:27:54.732779    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:27:55.644966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:55.644927    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 00:27:55.645169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:55.644984    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 00:27:55.645796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:55.645770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:55.736455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:55.729067    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 00:27:55.736455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:55.729137    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 00:27:55.931530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:55.929597    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:27:56.005138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:56.005047    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:56.117982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:56.117936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:56.645877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:56.645843    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 00:27:56.646390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:56.646361    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 00:27:56.727842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:56.727746    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 00:27:56.727842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:56.727808    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 00:27:57.488586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.488541    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:57.645449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.645403    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Nov 03 00:27:57.645624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.645482    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Nov 03 00:27:57.728519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.728477    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 00:27:57.728916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.728549    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 00:27:57.890642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.890444    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:57.986456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:57.982961    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:58.650675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:58.650637    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:27:58.652123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:58.652092    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:27:58.727703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:58.727666    4550 patch_prober.go:29] interesting pod/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Nov 03 00:27:58.727880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:58.727730    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Nov 03 00:27:59.199678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:59.199632    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:59.437425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:59.437389    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:27:59.816565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:59.816534    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:27:59.817847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:59.817824    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:27:59.818998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:27:59.818973    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:28:00.307619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:00.307586    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:01.165993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:01.165957    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:03.102884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:03.102844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:03.232197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:03.232164    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:03.342454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:03.339015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:04.074751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.074720    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:04.333645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.333558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:04.520698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.520669    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"]
Nov 03 00:28:04.520890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.520869    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy" containerID="cri-o://a6d3249a010a053a863d267bce2abdfef683e0ffb3dccec815180c73894d7705" gracePeriod=30
Nov 03 00:28:04.533899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.533868    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"]
Nov 03 00:28:04.534257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.534099    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy" containerID="cri-o://7ce606b7131cf57a5025ba4fdb52858411c7163c59e14ae0fcffb91f2318ad60" gracePeriod=30
Nov 03 00:28:04.541492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.539606    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"]
Nov 03 00:28:04.541492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.539824    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy" containerID="cri-o://897e1d3c25bd295e5a4c695279c494611a15dba3c90304dcbc9535cd17681306" gracePeriod=30
Nov 03 00:28:04.690763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.690724    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"]
Nov 03 00:28:04.691378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:04.691346    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" containerID="cri-o://62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c" gracePeriod=30
Nov 03 00:28:06.011783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:06.011750    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:06.112904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:06.112870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:06.213160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:06.213124    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:07.838362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:07.838329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:07.917211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:07.917176    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:07.960911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:07.960873    4550 patch_prober.go:29] interesting pod/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:28:07.961115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:07.960942    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:28:08.028460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:08.028371    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:09.685925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.685670    4550 generic.go:356] "Generic (PLEG): container finished" podID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerID="897e1d3c25bd295e5a4c695279c494611a15dba3c90304dcbc9535cd17681306" exitCode=0
Nov 03 00:28:09.685925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.685838    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" event={"ID":"2dd539aa-4521-49cc-9c0a-69fc213edceb","Type":"ContainerDied","Data":"897e1d3c25bd295e5a4c695279c494611a15dba3c90304dcbc9535cd17681306"}
Nov 03 00:28:09.688761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.688686    4550 generic.go:356] "Generic (PLEG): container finished" podID="12249439-f90c-4cb0-a78b-592da71491be" containerID="7ce606b7131cf57a5025ba4fdb52858411c7163c59e14ae0fcffb91f2318ad60" exitCode=0
Nov 03 00:28:09.688970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.688866    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" event={"ID":"12249439-f90c-4cb0-a78b-592da71491be","Type":"ContainerDied","Data":"7ce606b7131cf57a5025ba4fdb52858411c7163c59e14ae0fcffb91f2318ad60"}
Nov 03 00:28:09.691218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.691188    4550 generic.go:356] "Generic (PLEG): container finished" podID="b10124a4-efdf-408b-9602-6b9ff745217d" containerID="a6d3249a010a053a863d267bce2abdfef683e0ffb3dccec815180c73894d7705" exitCode=0
Nov 03 00:28:09.691509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.691296    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" event={"ID":"b10124a4-efdf-408b-9602-6b9ff745217d","Type":"ContainerDied","Data":"a6d3249a010a053a863d267bce2abdfef683e0ffb3dccec815180c73894d7705"}
Nov 03 00:28:09.826531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.826219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:09.934624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.934154    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:28:09.971412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:09.971380    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:28:10.000729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.000303    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:28:10.030921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.030824    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:10.051073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051040    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051121    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051164    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051184    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051237    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051257    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051297    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051315    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051334    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051357    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051381    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051402    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051420    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051515    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051534    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051566    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051604    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051628    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051647    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051682    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051700    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051717    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln\") pod \"2dd539aa-4521-49cc-9c0a-69fc213edceb\" (UID: \"2dd539aa-4521-49cc-9c0a-69fc213edceb\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051743    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051768    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051817    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket\") pod \"12249439-f90c-4cb0-a78b-592da71491be\" (UID: \"12249439-f90c-4cb0-a78b-592da71491be\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051841    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.051973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.051881    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token\") pod \"b10124a4-efdf-408b-9602-6b9ff745217d\" (UID: \"b10124a4-efdf-408b-9602-6b9ff745217d\") "
Nov 03 00:28:10.053287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.052746    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.053287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.052765    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.053287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.053047    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.053287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.053111    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:10.055766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.055735    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.056033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.055984    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.056685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.056659    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.057137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.056791    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.057137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.056835    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:10.059108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.058503    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.062061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.060885    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.062061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.061231    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:10.062391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.062295    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.062803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.062767    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.063548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.063421    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "kube-api-access-stpxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.065344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.065303    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:10.067098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.065551    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.067098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.065720    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.067281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.067042    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.067589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.067550    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:10.067688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.067665    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "kube-api-access-6xsk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.077185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.077084    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.077185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.077080    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.077915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.077885    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy" pod "b10124a4-efdf-408b-9602-6b9ff745217d" (UID: "b10124a4-efdf-408b-9602-6b9ff745217d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.078776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.078752    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "kube-api-access-nrlln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.078936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.078911    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token" pod "12249439-f90c-4cb0-a78b-592da71491be" (UID: "12249439-f90c-4cb0-a78b-592da71491be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.079359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.079338    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo" pod "2dd539aa-4521-49cc-9c0a-69fc213edceb" (UID: "2dd539aa-4521-49cc-9c0a-69fc213edceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:10.154324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154279    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154322    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154336    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154349    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154362    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b10124a4-efdf-408b-9602-6b9ff745217d-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154373    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154384    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154395    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154406    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154419    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154466    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd539aa-4521-49cc-9c0a-69fc213edceb-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154478    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154489    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154501    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stpxs\" (UniqueName: \"kubernetes.io/projected/12249439-f90c-4cb0-a78b-592da71491be-kube-api-access-stpxs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154515    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154527    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b10124a4-efdf-408b-9602-6b9ff745217d-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154537    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154548    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154561    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd539aa-4521-49cc-9c0a-69fc213edceb-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154573    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrlln\" (UniqueName: \"kubernetes.io/projected/2dd539aa-4521-49cc-9c0a-69fc213edceb-kube-api-access-nrlln\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154586    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12249439-f90c-4cb0-a78b-592da71491be-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154599    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154613    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12249439-f90c-4cb0-a78b-592da71491be-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154625    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b10124a4-efdf-408b-9602-6b9ff745217d-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.154628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154637    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.155350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154651    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsk5\" (UniqueName: \"kubernetes.io/projected/b10124a4-efdf-408b-9602-6b9ff745217d-kube-api-access-6xsk5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.155350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.154665    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12249439-f90c-4cb0-a78b-592da71491be-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.246028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.245994    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:10.410194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.410155    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:28:10.420409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.420373    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:10.460883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.460850    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.460933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461013    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461056    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461113    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461197    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461296    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:10.461415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461325    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461362    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461444    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data\") pod \"9d23d244-426e-4e20-a64e-ab3ff625ff00\" (UID: \"9d23d244-426e-4e20-a64e-ab3ff625ff00\") "
Nov 03 00:28:10.461678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461647    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.461893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.461870    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.462109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.462085    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.462109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.462100    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.462227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.462119    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d23d244-426e-4e20-a64e-ab3ff625ff00-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.462227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.462131    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.462227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.462145    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.464485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.464402    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:10.464612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.464582    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "kube-api-access-zpm6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.464612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.464602    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:10.464847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.464823    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo" pod "9d23d244-426e-4e20-a64e-ab3ff625ff00" (UID: "9d23d244-426e-4e20-a64e-ab3ff625ff00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:10.525293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.525255    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:10.563610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.563579    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.563610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.563613    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.563779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.563627    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d23d244-426e-4e20-a64e-ab3ff625ff00-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.563779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.563641    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpm6j\" (UniqueName: \"kubernetes.io/projected/9d23d244-426e-4e20-a64e-ab3ff625ff00-kube-api-access-zpm6j\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.563779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.563660    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.563779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.563673    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d23d244-426e-4e20-a64e-ab3ff625ff00-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:10.701732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.701701    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"
Nov 03 00:28:10.701732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.701713    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45" event={"ID":"2dd539aa-4521-49cc-9c0a-69fc213edceb","Type":"ContainerDied","Data":"2a6ae3920e6f6a04b00cf8cf1e77039920e67205c9bfc02975f5f9fe99c4d044"}
Nov 03 00:28:10.702243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.701814    4550 scope.go:122] "RemoveContainer" containerID="897e1d3c25bd295e5a4c695279c494611a15dba3c90304dcbc9535cd17681306"
Nov 03 00:28:10.703827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.703792    4550 generic.go:356] "Generic (PLEG): container finished" podID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerID="62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c" exitCode=0
Nov 03 00:28:10.703988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.703910    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" event={"ID":"9d23d244-426e-4e20-a64e-ab3ff625ff00","Type":"ContainerDied","Data":"62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c"}
Nov 03 00:28:10.703988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.703953    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65" event={"ID":"9d23d244-426e-4e20-a64e-ab3ff625ff00","Type":"ContainerDied","Data":"d44d61b12f7fee7fca0d15e6a390fe78a10d5efd233abffa2f0382aa57ef0374"}
Nov 03 00:28:10.704117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.704027    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"
Nov 03 00:28:10.707068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.707046    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"
Nov 03 00:28:10.707182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.707068    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g" event={"ID":"12249439-f90c-4cb0-a78b-592da71491be","Type":"ContainerDied","Data":"d32684d1b84f53d7727e08ffec00961d6ff372ea05d8251aa0028cd7799b1d05"}
Nov 03 00:28:10.708938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.708878    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6" event={"ID":"b10124a4-efdf-408b-9602-6b9ff745217d","Type":"ContainerDied","Data":"b47e19e0157d30cacb1247fbbb3797f3d92f61dec1a6b47640d928eb1b101a8c"}
Nov 03 00:28:10.709044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.708972    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"
Nov 03 00:28:10.715260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.715237    4550 scope.go:122] "RemoveContainer" containerID="62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c"
Nov 03 00:28:10.725570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.725550    4550 scope.go:122] "RemoveContainer" containerID="889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"
Nov 03 00:28:10.737631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.736914    4550 scope.go:122] "RemoveContainer" containerID="62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c"
Nov 03 00:28:10.738928 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:28:10.738894    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c\": container with ID starting with 62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c not found: ID does not exist" containerID="62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c"
Nov 03 00:28:10.739026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.738936    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c"} err="failed to get container status \"62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c\": rpc error: code = NotFound desc = could not find container \"62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c\": container with ID starting with 62b9333137c47e47c6f7fb988c69d741224a03293eb46f308c682d8857b0d24c not found: ID does not exist"
Nov 03 00:28:10.739026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.738954    4550 scope.go:122] "RemoveContainer" containerID="889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"
Nov 03 00:28:10.739198 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:28:10.739180    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9\": container with ID starting with 889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9 not found: ID does not exist" containerID="889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"
Nov 03 00:28:10.739272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.739209    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9"} err="failed to get container status \"889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9\": rpc error: code = NotFound desc = could not find container \"889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9\": container with ID starting with 889e688d3db17cc8e6e08aee34539daa0ba61fe200f75aad7e2bf0452d5a09c9 not found: ID does not exist"
Nov 03 00:28:10.739272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.739231    4550 scope.go:122] "RemoveContainer" containerID="7ce606b7131cf57a5025ba4fdb52858411c7163c59e14ae0fcffb91f2318ad60"
Nov 03 00:28:10.749028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.748993    4550 scope.go:122] "RemoveContainer" containerID="a6d3249a010a053a863d267bce2abdfef683e0ffb3dccec815180c73894d7705"
Nov 03 00:28:10.803901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.803868    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"]
Nov 03 00:28:10.816518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.816490    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-xqm45"]
Nov 03 00:28:10.853052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.853019    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"]
Nov 03 00:28:10.856118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.856092    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-q497g"]
Nov 03 00:28:10.889691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.889657    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"]
Nov 03 00:28:10.899047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.899017    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-bmhk6"]
Nov 03 00:28:10.934859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.934828    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"]
Nov 03 00:28:10.938219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:10.938189    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-27490b-openshift-default-6bc89c8cbf-vsc65"]
Nov 03 00:28:11.192059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.192024    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:11.600671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.600388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:11.739721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.739645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:11.809415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.809384    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12249439-f90c-4cb0-a78b-592da71491be" path="/var/lib/kubelet/pods/12249439-f90c-4cb0-a78b-592da71491be/volumes"
Nov 03 00:28:11.809885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.809870    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" path="/var/lib/kubelet/pods/2dd539aa-4521-49cc-9c0a-69fc213edceb/volumes"
Nov 03 00:28:11.810279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.810267    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" path="/var/lib/kubelet/pods/9d23d244-426e-4e20-a64e-ab3ff625ff00/volumes"
Nov 03 00:28:11.811044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.811029    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" path="/var/lib/kubelet/pods/b10124a4-efdf-408b-9602-6b9ff745217d/volumes"
Nov 03 00:28:11.856967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.856878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:11.953451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:11.953030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:12.092938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:12.092862    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:12.545500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:12.545464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:13.207456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:13.199326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:13.290547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:13.290511    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:14.281005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:14.280963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:16.745904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:16.745865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:16.849689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:16.849656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:17.035404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:17.035319    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:17.150182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:17.150147    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:18.311138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.311098    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:18.445264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.445231    4550 scope.go:122] "RemoveContainer" containerID="c0895d8d19a1ea78b946a09c50b8461186ed9ce34020338f8dd2d3c58c284b55"
Nov 03 00:28:18.454972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.454952    4550 scope.go:122] "RemoveContainer" containerID="8daeb0b4d7d716b8ef2243b1f0882d9365c6b9eb6368c89d4fceba86098e6456"
Nov 03 00:28:18.465265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.465241    4550 scope.go:122] "RemoveContainer" containerID="eddd7b61aa4b1aebf5e96e5a9025f0ce1953f6158f6899ad08c83a64111205ff"
Nov 03 00:28:18.470045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.470000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:18.567633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.567543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:18.689208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.689164    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:18.779655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:18.779623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:20.729872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:20.729835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:22.415769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:22.415735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:22.963680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:22.963639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:24.266329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:24.266282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:24.726118 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:28:24.726075    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:28:26.167970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.167934    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:26.268291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.268245    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"]
Nov 03 00:28:26.268564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.268537    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc" podUID="f474dc7f-810f-4ebf-9518-6aa5df9e994f" containerName="payload-processing" containerID="cri-o://091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20" gracePeriod=30
Nov 03 00:28:26.341740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.341688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:26.445105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.444987    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:26.755497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.755124    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:28:26.835598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.835569    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume\") pod \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") "
Nov 03 00:28:26.835781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.835668    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789\" (UniqueName: \"kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789\") pod \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\" (UID: \"f474dc7f-810f-4ebf-9518-6aa5df9e994f\") "
Nov 03 00:28:26.836026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.836005    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume" pod "f474dc7f-810f-4ebf-9518-6aa5df9e994f" (UID: "f474dc7f-810f-4ebf-9518-6aa5df9e994f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:26.836314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.836291    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f474dc7f-810f-4ebf-9518-6aa5df9e994f-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:26.838330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.838298    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789" pod "f474dc7f-810f-4ebf-9518-6aa5df9e994f" (UID: "f474dc7f-810f-4ebf-9518-6aa5df9e994f"). InnerVolumeSpecName "kube-api-access-6s789". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:26.839924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.839898    4550 generic.go:356] "Generic (PLEG): container finished" podID="f474dc7f-810f-4ebf-9518-6aa5df9e994f" containerID="091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20" exitCode=0
Nov 03 00:28:26.840037 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.839945    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"
Nov 03 00:28:26.840037 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.839974    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc" event={"ID":"f474dc7f-810f-4ebf-9518-6aa5df9e994f","Type":"ContainerDied","Data":"091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20"}
Nov 03 00:28:26.840120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.840035    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc" event={"ID":"f474dc7f-810f-4ebf-9518-6aa5df9e994f","Type":"ContainerDied","Data":"b7823232778316178ce1dcd1dd327156c37457b42e94c4723becbca3714e9e00"}
Nov 03 00:28:26.840120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.840065    4550 scope.go:122] "RemoveContainer" containerID="091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20"
Nov 03 00:28:26.860202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.860177    4550 scope.go:122] "RemoveContainer" containerID="091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20"
Nov 03 00:28:26.860630 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:28:26.860599    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20\": container with ID starting with 091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20 not found: ID does not exist" containerID="091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20"
Nov 03 00:28:26.860728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.860639    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20"} err="failed to get container status \"091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20\": rpc error: code = NotFound desc = could not find container \"091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20\": container with ID starting with 091ea20cff52373e47b94cdaf680afa459770610d0701e67e3f4c22f09c7af20 not found: ID does not exist"
Nov 03 00:28:26.897220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.897188    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"]
Nov 03 00:28:26.901671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.901643    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-41e815-6bf7cffc54-wjrxc"]
Nov 03 00:28:26.937107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:26.937065    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s789\" (UniqueName: \"kubernetes.io/projected/f474dc7f-810f-4ebf-9518-6aa5df9e994f-kube-api-access-6s789\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:27.030723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:27.030637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:27.808500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:27.808466    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f474dc7f-810f-4ebf-9518-6aa5df9e994f" path="/var/lib/kubelet/pods/f474dc7f-810f-4ebf-9518-6aa5df9e994f/volumes"
Nov 03 00:28:28.461452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:28.461217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:29.059993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:29.059954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:29.151098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:29.151065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:29.321371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:29.321282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:29.762879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:29.762844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:30.150237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:30.150205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:30.912256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:30.912216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:31.450804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:31.450764    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:33.443991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:33.443952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:33.649304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:33.649266    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:33.749567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:33.749480    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:34.387985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:34.387951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:34.739954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:34.739910    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"]
Nov 03 00:28:34.740505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:34.740149    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy" containerID="cri-o://27eda38831f13dda4d048cd8e0fdc419614caec62f4cba0773aa44a1c0f4a3fa" gracePeriod=30
Nov 03 00:28:34.766005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:34.765956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:35.477310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:35.477276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:35.672971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:35.672931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:36.167036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:36.166854    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:36.268955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:36.268911    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:36.367503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:36.367471    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:37.253593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:37.253562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:37.891585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:37.891549    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:38.125001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:38.124969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:38.223209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:38.223166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:39.962852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:39.962566    4550 generic.go:356] "Generic (PLEG): container finished" podID="15f56704-a672-4db5-acd4-3708cef007c8" containerID="27eda38831f13dda4d048cd8e0fdc419614caec62f4cba0773aa44a1c0f4a3fa" exitCode=0
Nov 03 00:28:39.962852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:39.962665    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" event={"ID":"15f56704-a672-4db5-acd4-3708cef007c8","Type":"ContainerDied","Data":"27eda38831f13dda4d048cd8e0fdc419614caec62f4cba0773aa44a1c0f4a3fa"}
Nov 03 00:28:39.962852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:39.962758    4550 scope.go:122] "RemoveContainer" containerID="3e9b5987ba525891adc2af7b67206d73e4fc0770222243dfb02b107511c53c4a"
Nov 03 00:28:40.058412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.058377    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:28:40.080956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.080927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:40.172297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172261    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172368    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172396    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172463    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172509    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172543    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172567    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172596    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.172742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.172637    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data\") pod \"15f56704-a672-4db5-acd4-3708cef007c8\" (UID: \"15f56704-a672-4db5-acd4-3708cef007c8\") "
Nov 03 00:28:40.173664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.173154    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:40.173664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.173371    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:40.173664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.173423    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:40.173664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.173605    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:40.173664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.173625    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:40.175869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.175831    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:40.176289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.176259    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:40.176790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.176751    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "kube-api-access-85k6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:40.176883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.176841    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token" pod "15f56704-a672-4db5-acd4-3708cef007c8" (UID: "15f56704-a672-4db5-acd4-3708cef007c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275011    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15f56704-a672-4db5-acd4-3708cef007c8-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275042    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275052    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275061    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275070    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275078    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15f56704-a672-4db5-acd4-3708cef007c8-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275086    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85k6f\" (UniqueName: \"kubernetes.io/projected/15f56704-a672-4db5-acd4-3708cef007c8-kube-api-access-85k6f\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275095    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.275100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.275104    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15f56704-a672-4db5-acd4-3708cef007c8-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:40.730838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.730799    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:40.836603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.836568    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:40.970665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.970634    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"
Nov 03 00:28:40.970665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.970637    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq" event={"ID":"15f56704-a672-4db5-acd4-3708cef007c8","Type":"ContainerDied","Data":"465a2d61861adef603bdd47831889ecfd66106d3d53127857a8ac88ba2573e7d"}
Nov 03 00:28:40.971289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:40.970713    4550 scope.go:122] "RemoveContainer" containerID="27eda38831f13dda4d048cd8e0fdc419614caec62f4cba0773aa44a1c0f4a3fa"
Nov 03 00:28:41.038413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:41.038375    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"]
Nov 03 00:28:41.051391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:41.051360    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1d29dd-openshift-default-65d4b58fdc-sjcrq"]
Nov 03 00:28:41.808295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:41.808260    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f56704-a672-4db5-acd4-3708cef007c8" path="/var/lib/kubelet/pods/15f56704-a672-4db5-acd4-3708cef007c8/volumes"
Nov 03 00:28:41.912954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:41.912921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:42.454222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:42.454183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:42.555289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:42.555256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:43.982491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:43.982421    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:45.027387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:45.027354    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"]
Nov 03 00:28:45.041903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:45.041876    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a2ed4e7-6npd2"]
Nov 03 00:28:45.253311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:45.253272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:45.346443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:45.346305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:45.446833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:45.446793    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:45.885264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:45.885222    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e69467e-9fb6-4734-aa70-26060ad42d31" path="/var/lib/kubelet/pods/0e69467e-9fb6-4734-aa70-26060ad42d31/volumes"
Nov 03 00:28:46.028011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:46.027972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:47.584560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:47.584527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:47.807847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:47.807815    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:47.902060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:47.902030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:48.473932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:48.473893    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:49.028818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:49.028782    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"]
Nov 03 00:28:49.040200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:49.040164    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90ca748b-pxg5k"]
Nov 03 00:28:49.810367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:49.810333    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15f42e6-cc9b-4524-99c0-a2552aad1d28" path="/var/lib/kubelet/pods/a15f42e6-cc9b-4524-99c0-a2552aad1d28/volumes"
Nov 03 00:28:50.173900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:50.173867    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:50.987913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:50.987865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:51.372575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:51.372477    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:51.541911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:51.541281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:51.747086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:51.747039    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:52.541231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:52.541186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:53.229324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:53.229293    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:53.329235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:53.329201    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:53.996457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:53.996393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:54.348592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:54.348515    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"]
Nov 03 00:28:54.349047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:54.349022    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" containerID="cri-o://5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695" gracePeriod=30
Nov 03 00:28:54.369593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:54.369516    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:54.601104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:54.600952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:54.727735 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:28:54.727695    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:28:55.880538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:55.880502    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:56.266910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:56.266877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:56.363905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:56.363879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:56.491780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:56.491739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:58.542637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:58.542604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:28:58.654069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:58.654028    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:28:58.654264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:58.654101    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:28:59.639113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.639085    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:28:59.720081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720043    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720094    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720174    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720213    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720239    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720280    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720332    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720358    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720514    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket\") pod \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\" (UID: \"55b1cea3-144f-40c0-91c9-12f1d3ff93ec\") "
Nov 03 00:28:59.720738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720614    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:59.720738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720630    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:59.720840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720814    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:59.720946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.720924    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:59.721365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.721095    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:59.721365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.721269    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.721365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.721288    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.721365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.721301    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.721365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.721313    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.721365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.721326    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.722950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.722926    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:59.724250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.724221    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:59.724392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.724366    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "kube-api-access-jwnx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:59.724923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.724906    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token" pod "55b1cea3-144f-40c0-91c9-12f1d3ff93ec" (UID: "55b1cea3-144f-40c0-91c9-12f1d3ff93ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:59.823006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.822978    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.823006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.823007    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnx4\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-kube-api-access-jwnx4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.823169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.823018    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:28:59.823169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:28:59.823027    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55b1cea3-144f-40c0-91c9-12f1d3ff93ec-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:29:00.177742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.177702    4550 generic.go:356] "Generic (PLEG): container finished" podID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerID="5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695" exitCode=0
Nov 03 00:29:00.178142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.177853    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" event={"ID":"55b1cea3-144f-40c0-91c9-12f1d3ff93ec","Type":"ContainerDied","Data":"5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695"}
Nov 03 00:29:00.180010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.178486    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68" event={"ID":"55b1cea3-144f-40c0-91c9-12f1d3ff93ec","Type":"ContainerDied","Data":"1383fd027ec12657acc8a41c8b5c62bb116cdcb2116ff568eaa1a899cf771c34"}
Nov 03 00:29:00.180205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.177954    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"
Nov 03 00:29:00.182035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.179824    4550 scope.go:122] "RemoveContainer" containerID="5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695"
Nov 03 00:29:00.196558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.196449    4550 scope.go:122] "RemoveContainer" containerID="79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"
Nov 03 00:29:00.208038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.207940    4550 scope.go:122] "RemoveContainer" containerID="5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695"
Nov 03 00:29:00.208503 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:29:00.208479    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695\": container with ID starting with 5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695 not found: ID does not exist" containerID="5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695"
Nov 03 00:29:00.208698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.208667    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695"} err="failed to get container status \"5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695\": rpc error: code = NotFound desc = could not find container \"5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695\": container with ID starting with 5a679f73e66481a76f6c6e1eed2df086f604ce0238670c8373105a30cefa4695 not found: ID does not exist"
Nov 03 00:29:00.208698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.208698    4550 scope.go:122] "RemoveContainer" containerID="79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"
Nov 03 00:29:00.209047 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:29:00.209019    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39\": container with ID starting with 79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39 not found: ID does not exist" containerID="79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"
Nov 03 00:29:00.209159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.209056    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39"} err="failed to get container status \"79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39\": rpc error: code = NotFound desc = could not find container \"79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39\": container with ID starting with 79ab2b7997cde6517884e30faf3fee8822c21c8f2fe5f440f3e9e0b5300f9c39 not found: ID does not exist"
Nov 03 00:29:00.248975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.248939    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:00.251056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.251032    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"]
Nov 03 00:29:00.265970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.265933    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dda6df-openshift-default-6c99d4898c-njh68"]
Nov 03 00:29:00.353528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.353482    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:00.617789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.617688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:00.726587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.726558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:00.972995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:00.972962    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:01.809905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:01.809868    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" path="/var/lib/kubelet/pods/55b1cea3-144f-40c0-91c9-12f1d3ff93ec/volumes"
Nov 03 00:29:02.364445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:02.364394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:02.471187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:02.471154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:02.909640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:02.909601    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:03.084819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:03.084743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:03.184217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:03.184125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:03.388229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:03.388191    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:05.259921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:05.259882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:06.538988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:06.538938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:06.738637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:06.738580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:07.717967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:07.717931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:09.176715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:09.176671    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:09.495235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:09.495142    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:10.147930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:10.147895    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:11.162677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:11.162642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:12.990018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:12.989984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:15.006162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:15.006113    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:15.347315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:15.347220    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:17.140178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:17.140133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:18.527639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:18.527609    4550 scope.go:122] "RemoveContainer" containerID="cb3a9521e0e5ba4117e652c1dcab87b002b596f16b9dd42c6a87a4d69909d566"
Nov 03 00:29:18.659038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:18.659000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:18.845093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:18.845005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:18.953927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:18.953889    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:19.669545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:19.669497    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:20.920866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:20.920830    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:21.095717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:21.095680    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:21.520457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:21.519420    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:21.617037 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:21.616999    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:22.295398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:22.295365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:22.395181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:22.395145    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:22.497528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:22.497482    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:22.674932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:22.674900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:23.121769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:23.121672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:23.234285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:23.234251    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:23.485657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:23.485624    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:24.529630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:24.529599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:24.723174 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:29:24.723137    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:29:26.491561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:26.491505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:27.117223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:27.117193    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:27.268663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:27.268625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:28.390499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:28.390411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:28.826531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:28.826492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:28.933812 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:28.933778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:30.188189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:30.188144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:31.181043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:31.181005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:31.707400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:31.707363    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:31.802995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:31.802960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:31.904016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:31.903984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:33.024350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:33.024311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:34.686204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:34.686173    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:36.159751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:36.159719    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:36.339623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:36.339591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:36.435771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:36.435678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:36.980337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:36.980297    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:38.694224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:38.694185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:39.988403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:39.988367    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:40.468700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:40.468649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:40.581468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:40.581415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:40.703103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:40.703073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:42.838296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:42.838267    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:43.760880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:43.760846    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:43.871013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:43.870976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:44.780668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:44.780637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:44.879855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:44.879822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:44.981973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:44.981941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:45.806548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:45.806513    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:46.020050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:46.020016    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:46.381252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:46.381220    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:46.507214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:46.507167    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:46.788564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:46.788529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:47.125398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:47.125321    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:47.584505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:47.584472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:47.698072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:47.698032    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:48.200264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:48.200227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:48.309592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:48.309561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:48.415446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:48.415396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:49.256101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:49.256043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:49.711950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:49.711921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:49.878401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:49.878325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:50.420442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:50.420396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:50.933896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:50.933865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:51.070715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:51.070676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:51.172801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:51.172762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:52.827472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.827425    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:52.927364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.927328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:52.945277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945242    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"]
Nov 03 00:29:52.945850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945830    4550 cpu_manager.go:414] "containerMap: removing container" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945850    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945874    4550 cpu_manager.go:414] "containerMap: removing container" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945882    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945909    4550 cpu_manager.go:414] "containerMap: removing container" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945917    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945931    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945938    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945956    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy"
Nov 03 00:29:52.945966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945963    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945973    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945979    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945991    4550 cpu_manager.go:414] "containerMap: removing container" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.945998    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946011    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946018    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946032    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f474dc7f-810f-4ebf-9518-6aa5df9e994f" containerName="payload-processing"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946039    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f474dc7f-810f-4ebf-9518-6aa5df9e994f" containerName="payload-processing"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946180    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946195    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10124a4-efdf-408b-9602-6b9ff745217d" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946205    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946212    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d23d244-426e-4e20-a64e-ab3ff625ff00" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946221    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946229    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946238    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd539aa-4521-49cc-9c0a-69fc213edceb" containerName="istio-proxy"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946248    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f474dc7f-810f-4ebf-9518-6aa5df9e994f" containerName="payload-processing"
Nov 03 00:29:52.946419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.946258    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="12249439-f90c-4cb0-a78b-592da71491be" containerName="istio-proxy"
Nov 03 00:29:52.950010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.949989    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:52.968381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.968351    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"]
Nov 03 00:29:52.977376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:52.977358    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-191362\""
Nov 03 00:29:53.148504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.146547    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzfv\" (UniqueName: \"kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv\") pod \"payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.148504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.146635    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.248088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.248057    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.248333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.248312    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzfv\" (UniqueName: \"kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv\") pod \"payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.248850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.248834    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.269156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.269119    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzfv\" (UniqueName: \"kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv\") pod \"payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.565058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.565021    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:53.859879 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:29:53.859845    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebfc60c6_26a6_45d4_9450_906e7e1652e1.slice/crio-e1b969e2eb3ee1e6f9cd4916e6a9da9cb87a8714f9222ec87024cf8438962e8a WatchSource:0}: Error finding container e1b969e2eb3ee1e6f9cd4916e6a9da9cb87a8714f9222ec87024cf8438962e8a: Status 404 returned error can't find the container with id e1b969e2eb3ee1e6f9cd4916e6a9da9cb87a8714f9222ec87024cf8438962e8a
Nov 03 00:29:53.861664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.861642    4550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:29:53.862749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:53.862723    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"]
Nov 03 00:29:54.033165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:54.033121    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:54.138995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:54.138924    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:54.588139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:54.588098    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7" event={"ID":"ebfc60c6-26a6-45d4-9450-906e7e1652e1","Type":"ContainerStarted","Data":"e1b969e2eb3ee1e6f9cd4916e6a9da9cb87a8714f9222ec87024cf8438962e8a"}
Nov 03 00:29:54.730615 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:29:54.730577    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:29:55.515460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:55.515384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:55.768286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:55.768212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:55.876983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:55.876952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:56.601660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:56.601626    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7" event={"ID":"ebfc60c6-26a6-45d4-9450-906e7e1652e1","Type":"ContainerStarted","Data":"7b1f0958323fa962130f72861c0cd610b76603fb7847272144803d06bf05db71"}
Nov 03 00:29:56.602086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:56.601717    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:29:56.643880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:56.643836    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7" podStartSLOduration=2.539424397 podStartE2EDuration="4.643823911s" totalImagesPullingTime="2.104399514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:56.642751462 +0000 UTC m=+1478.964288291" watchObservedRunningTime="2026-11-03 00:29:56.643823911 +0000 UTC m=+1478.965360751"
Nov 03 00:29:57.460542 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:57.460506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:29:59.125865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:29:59.125836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:00.164423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.164331    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"]
Nov 03 00:30:00.165108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.165089    4550 cpu_manager.go:414] "containerMap: removing container" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy"
Nov 03 00:30:00.165184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.165110    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f56704-a672-4db5-acd4-3708cef007c8" containerName="istio-proxy"
Nov 03 00:30:00.165246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.165235    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b1cea3-144f-40c0-91c9-12f1d3ff93ec" containerName="istio-proxy"
Nov 03 00:30:00.169061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.169038    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"]
Nov 03 00:30:00.169296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.169269    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.173573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.173554    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"]
Nov 03 00:30:00.173702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.173683    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.179747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.179730    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.191596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.191577    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"]
Nov 03 00:30:00.199674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.199629    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:30:00.201710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.201658    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.218030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.217994    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"]
Nov 03 00:30:00.247708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.247683    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258167    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98lgs\" (UniqueName: \"kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258230    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-cr6gl\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258296    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7mr\" (UniqueName: \"kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr\") pod \"maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258369    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258448    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258500    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258559    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw7p7\" (UniqueName: \"kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7\") pod \"maas-api-key-cleanup-29894430-cr6gl\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.258627    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vz5\" (UniqueName: \"kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.265478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.264984    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"]
Nov 03 00:30:00.280094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.279633    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"]
Nov 03 00:30:00.280094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.279682    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"]
Nov 03 00:30:00.280094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.279708    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"]
Nov 03 00:30:00.280094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.279780    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.289810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.289787    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"]
Nov 03 00:30:00.292209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.292092    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"]
Nov 03 00:30:00.296511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.296492    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"]
Nov 03 00:30:00.359639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359603    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98lgs\" (UniqueName: \"kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.359795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359646    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-cr6gl\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.359795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359666    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nr2c\" (UniqueName: \"kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.359795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359727    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7mr\" (UniqueName: \"kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr\") pod \"maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.359795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359757    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lb4\" (UniqueName: \"kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.359795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359788    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.360056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359843    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.360056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.359898    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.360155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.360067    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw7p7\" (UniqueName: \"kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7\") pod \"maas-api-key-cleanup-29894430-cr6gl\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.360155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.360129    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.360266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.360183    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vz5\" (UniqueName: \"kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.360266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.360241    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894430-cr6gl\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.360368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.360287    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.360685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.360657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.361164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.361135    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.361265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.361168    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.380125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.380099    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7mr\" (UniqueName: \"kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr\") pod \"maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.380275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.380257    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vz5\" (UniqueName: \"kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.380748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.380729    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98lgs\" (UniqueName: \"kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.380897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.380878    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw7p7\" (UniqueName: \"kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7\") pod \"maas-api-key-cleanup-29894430-cr6gl\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.461737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.461671    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.461853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.461769    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nr2c\" (UniqueName: \"kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.461853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.461830    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lb4\" (UniqueName: \"kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.461975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.461958    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.462322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.462295    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.462744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.462723    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.482713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.482692    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lb4\" (UniqueName: \"kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.482927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.482907    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nr2c\" (UniqueName: \"kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c\") pod \"maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.486584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.486564    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:00.525463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.525413    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:00.548744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.547160    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:00.554152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.554129    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:00.590272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.583603    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:00.610846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.610812    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:00.793332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.793302    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"]
Nov 03 00:30:00.793843 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:30:00.793821    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9eeace3_0a7d_4f8b_8083_80ae7d80e6f5.slice/crio-0a7eaaf70ecc2039ba60facf652bb62105ae4dd535c81c410d3a37eb2368d48b WatchSource:0}: Error finding container 0a7eaaf70ecc2039ba60facf652bb62105ae4dd535c81c410d3a37eb2368d48b: Status 404 returned error can't find the container with id 0a7eaaf70ecc2039ba60facf652bb62105ae4dd535c81c410d3a37eb2368d48b
Nov 03 00:30:00.846304 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:30:00.846274    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b691a4_0be8_4942_b5d4_d7236acb1cb7.slice/crio-3b22860aa66119f535d1658dfaeac0578ae2123fd265c568202a55cd154f4291 WatchSource:0}: Error finding container 3b22860aa66119f535d1658dfaeac0578ae2123fd265c568202a55cd154f4291: Status 404 returned error can't find the container with id 3b22860aa66119f535d1658dfaeac0578ae2123fd265c568202a55cd154f4291
Nov 03 00:30:00.847133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.847108    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"]
Nov 03 00:30:00.909087 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:30:00.909059    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e53f58_dbc7_4da5_9815_84620740bc54.slice/crio-1ab3238895d7ebd104012543d5ff35e98826920c06cb21644dd282263ce1cc89 WatchSource:0}: Error finding container 1ab3238895d7ebd104012543d5ff35e98826920c06cb21644dd282263ce1cc89: Status 404 returned error can't find the container with id 1ab3238895d7ebd104012543d5ff35e98826920c06cb21644dd282263ce1cc89
Nov 03 00:30:00.910004 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:30:00.909977    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53ab800_74b9_4633_b863_c8129b2247f7.slice/crio-f9ba54320d3b13e41aa6115800b23b77dc3424529f80e851180398c81d82bae5 WatchSource:0}: Error finding container f9ba54320d3b13e41aa6115800b23b77dc3424529f80e851180398c81d82bae5: Status 404 returned error can't find the container with id f9ba54320d3b13e41aa6115800b23b77dc3424529f80e851180398c81d82bae5
Nov 03 00:30:00.913355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.913337    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"]
Nov 03 00:30:00.917181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.917157    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"]
Nov 03 00:30:00.939839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.939679    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"]
Nov 03 00:30:00.940458 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:30:00.940412    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26ea9ca_7b6a_4569_abe3_5523c98afd35.slice/crio-7ff2c192e561c5c6f3bc4af048d3d66df14ad2380f084086f2b2600353511376 WatchSource:0}: Error finding container 7ff2c192e561c5c6f3bc4af048d3d66df14ad2380f084086f2b2600353511376: Status 404 returned error can't find the container with id 7ff2c192e561c5c6f3bc4af048d3d66df14ad2380f084086f2b2600353511376
Nov 03 00:30:00.974611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.974533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:00.980992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:00.980754    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"]
Nov 03 00:30:01.642405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.642301    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerStarted","Data":"5de94d12a81d7824328ab84e392f8512ab54b3bde5a3447c96b77754c5238584"}
Nov 03 00:30:01.642405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.642388    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerStarted","Data":"c4d9cd3858d236174a87f02626a4c0067c370777d8f2dcb9f0b6b2efb100a4ea"}
Nov 03 00:30:01.644073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.644044    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerStarted","Data":"3bacb080be70c7ca0c8d0f131a7a74e69bedc976adec0085a6309cc56e03e3ac"}
Nov 03 00:30:01.644206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.644089    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerStarted","Data":"f9ba54320d3b13e41aa6115800b23b77dc3424529f80e851180398c81d82bae5"}
Nov 03 00:30:01.645540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.645514    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerStarted","Data":"f176ea02bbc0eb057fb1cb14e6a82402285f89008385222d5df84d3af085fbc2"}
Nov 03 00:30:01.645663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.645559    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerStarted","Data":"7ff2c192e561c5c6f3bc4af048d3d66df14ad2380f084086f2b2600353511376"}
Nov 03 00:30:01.647090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.647063    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerStarted","Data":"fb94602aa24a519765d393e7b0f99fdb28e3c896ee98fe6e963e34d58a0525af"}
Nov 03 00:30:01.647213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.647115    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerStarted","Data":"3b22860aa66119f535d1658dfaeac0578ae2123fd265c568202a55cd154f4291"}
Nov 03 00:30:01.648550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.648528    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerStarted","Data":"1b57b31c0dd3d649c27394e6a990795bef92b1b1676860bfe58070d782bdd710"}
Nov 03 00:30:01.648669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.648562    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerStarted","Data":"0a7eaaf70ecc2039ba60facf652bb62105ae4dd535c81c410d3a37eb2368d48b"}
Nov 03 00:30:01.649963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.649941    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerStarted","Data":"23246472e01daeea67246dc043a9f166d5f2c16314a9588dd000637ada733e03"}
Nov 03 00:30:01.650071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.649975    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerStarted","Data":"1ab3238895d7ebd104012543d5ff35e98826920c06cb21644dd282263ce1cc89"}
Nov 03 00:30:01.686041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.685997    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" podStartSLOduration=1.685984543 podStartE2EDuration="1.685984543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.68264388 +0000 UTC m=+1484.004180679" watchObservedRunningTime="2026-11-03 00:30:01.685984543 +0000 UTC m=+1484.007521341"
Nov 03 00:30:01.722872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.722828    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" podStartSLOduration=1.722815668 podStartE2EDuration="1.722815668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.721333802 +0000 UTC m=+1484.042870617" watchObservedRunningTime="2026-11-03 00:30:01.722815668 +0000 UTC m=+1484.044352468"
Nov 03 00:30:01.763495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.763406    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" podStartSLOduration=1.763388134 podStartE2EDuration="1.763388134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.761087938 +0000 UTC m=+1484.082624736" watchObservedRunningTime="2026-11-03 00:30:01.763388134 +0000 UTC m=+1484.084924935"
Nov 03 00:30:01.800688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.800642    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" podStartSLOduration=1.80063034 podStartE2EDuration="1.80063034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.800579656 +0000 UTC m=+1484.122116457" watchObservedRunningTime="2026-11-03 00:30:01.80063034 +0000 UTC m=+1484.122167138"
Nov 03 00:30:01.840962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.840916    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" podStartSLOduration=1.840904111 podStartE2EDuration="1.840904111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.838091821 +0000 UTC m=+1484.159628622" watchObservedRunningTime="2026-11-03 00:30:01.840904111 +0000 UTC m=+1484.162440909"
Nov 03 00:30:01.877127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:01.877084    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" podStartSLOduration=1.877072789 podStartE2EDuration="1.877072789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:01.876920651 +0000 UTC m=+1484.198457452" watchObservedRunningTime="2026-11-03 00:30:01.877072789 +0000 UTC m=+1484.198609587"
Nov 03 00:30:02.629730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.629683    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:02.658617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.658588    4550 generic.go:356] "Generic (PLEG): container finished" podID="d53ab800-74b9-4633-b863-c8129b2247f7" containerID="3bacb080be70c7ca0c8d0f131a7a74e69bedc976adec0085a6309cc56e03e3ac" exitCode=60
Nov 03 00:30:02.659040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.658680    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerDied","Data":"3bacb080be70c7ca0c8d0f131a7a74e69bedc976adec0085a6309cc56e03e3ac"}
Nov 03 00:30:02.659040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.658911    4550 scope.go:122] "RemoveContainer" containerID="3bacb080be70c7ca0c8d0f131a7a74e69bedc976adec0085a6309cc56e03e3ac"
Nov 03 00:30:02.660552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.660512    4550 generic.go:356] "Generic (PLEG): container finished" podID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerID="f176ea02bbc0eb057fb1cb14e6a82402285f89008385222d5df84d3af085fbc2" exitCode=60
Nov 03 00:30:02.660612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.660591    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerDied","Data":"f176ea02bbc0eb057fb1cb14e6a82402285f89008385222d5df84d3af085fbc2"}
Nov 03 00:30:02.660803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.660786    4550 scope.go:122] "RemoveContainer" containerID="f176ea02bbc0eb057fb1cb14e6a82402285f89008385222d5df84d3af085fbc2"
Nov 03 00:30:02.662529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.662366    4550 generic.go:356] "Generic (PLEG): container finished" podID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerID="fb94602aa24a519765d393e7b0f99fdb28e3c896ee98fe6e963e34d58a0525af" exitCode=60
Nov 03 00:30:02.662674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.662645    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerDied","Data":"fb94602aa24a519765d393e7b0f99fdb28e3c896ee98fe6e963e34d58a0525af"}
Nov 03 00:30:02.662751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.662715    4550 scope.go:122] "RemoveContainer" containerID="fb94602aa24a519765d393e7b0f99fdb28e3c896ee98fe6e963e34d58a0525af"
Nov 03 00:30:02.664333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.664307    4550 generic.go:356] "Generic (PLEG): container finished" podID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerID="1b57b31c0dd3d649c27394e6a990795bef92b1b1676860bfe58070d782bdd710" exitCode=60
Nov 03 00:30:02.664469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.664332    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerDied","Data":"1b57b31c0dd3d649c27394e6a990795bef92b1b1676860bfe58070d782bdd710"}
Nov 03 00:30:02.664584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.664568    4550 scope.go:122] "RemoveContainer" containerID="1b57b31c0dd3d649c27394e6a990795bef92b1b1676860bfe58070d782bdd710"
Nov 03 00:30:02.666214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.666193    4550 generic.go:356] "Generic (PLEG): container finished" podID="40e53f58-dbc7-4da5-9815-84620740bc54" containerID="23246472e01daeea67246dc043a9f166d5f2c16314a9588dd000637ada733e03" exitCode=60
Nov 03 00:30:02.666322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.666244    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerDied","Data":"23246472e01daeea67246dc043a9f166d5f2c16314a9588dd000637ada733e03"}
Nov 03 00:30:02.666534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.666517    4550 scope.go:122] "RemoveContainer" containerID="23246472e01daeea67246dc043a9f166d5f2c16314a9588dd000637ada733e03"
Nov 03 00:30:02.668054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.668024    4550 generic.go:356] "Generic (PLEG): container finished" podID="dad38199-026b-4603-8647-08f578e29c6f" containerID="5de94d12a81d7824328ab84e392f8512ab54b3bde5a3447c96b77754c5238584" exitCode=60
Nov 03 00:30:02.668148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.668056    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerDied","Data":"5de94d12a81d7824328ab84e392f8512ab54b3bde5a3447c96b77754c5238584"}
Nov 03 00:30:02.668289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:02.668273    4550 scope.go:122] "RemoveContainer" containerID="5de94d12a81d7824328ab84e392f8512ab54b3bde5a3447c96b77754c5238584"
Nov 03 00:30:03.678156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.678117    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerStarted","Data":"fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f"}
Nov 03 00:30:03.680242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.680215    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerStarted","Data":"722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c"}
Nov 03 00:30:03.682335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.682295    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerStarted","Data":"d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0"}
Nov 03 00:30:03.684342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.684317    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerStarted","Data":"c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e"}
Nov 03 00:30:03.686572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.686546    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerStarted","Data":"98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e"}
Nov 03 00:30:03.688616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.688588    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerStarted","Data":"a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6"}
Nov 03 00:30:03.802251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:03.802194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:04.206560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.206412    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"]
Nov 03 00:30:04.222911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.211733    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.233466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.228886    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b6cd31-openshift-default-dockercfg-r6rqb\""
Nov 03 00:30:04.248513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.243735    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"]
Nov 03 00:30:04.248513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.244856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:04.348927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.348661    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.348927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.348770    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.348927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.348849    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.348927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.348881    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.348927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.348915    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.349341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.348977    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.349341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.349056    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.349341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.349132    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhx6p\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.349341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.349172    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.375455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.372953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451177    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451234    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451257    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451308    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451377    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451460    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhx6p\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451496    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451572    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.451645    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.452466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.452382    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.453143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.452632    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.453143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.452861    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.462496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.453525    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.468447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.466297    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.488593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.482085    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.488593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.482872    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.502465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.494887    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.502465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.501396    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhx6p\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p\") pod \"e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") " pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.554601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.552069    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.705173    4550 generic.go:356] "Generic (PLEG): container finished" podID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerID="d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0" exitCode=60
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.705304    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerDied","Data":"d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0"}
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.705376    4550 scope.go:122] "RemoveContainer" containerID="fb94602aa24a519765d393e7b0f99fdb28e3c896ee98fe6e963e34d58a0525af"
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.705595    4550 scope.go:122] "RemoveContainer" containerID="d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0"
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:04.705913    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk_odh-ai-gateway-infra(b1b691a4-0be8-4942-b5d4-d7236acb1cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7"
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.709595    4550 generic.go:356] "Generic (PLEG): container finished" podID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerID="c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e" exitCode=60
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.709669    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerDied","Data":"c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e"}
Nov 03 00:30:04.710220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.710005    4550 scope.go:122] "RemoveContainer" containerID="c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e"
Nov 03 00:30:04.711242 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:04.710266    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-cr6gl_odh-ai-gateway-infra(e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5"
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.713340    4550 generic.go:356] "Generic (PLEG): container finished" podID="40e53f58-dbc7-4da5-9815-84620740bc54" containerID="98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e" exitCode=60
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.713407    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerDied","Data":"98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e"}
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.713712    4550 scope.go:122] "RemoveContainer" containerID="98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e"
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:04.713925    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6_odh-ai-gateway-infra(40e53f58-dbc7-4da5-9815-84620740bc54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" podUID="40e53f58-dbc7-4da5-9815-84620740bc54"
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.715782    4550 generic.go:356] "Generic (PLEG): container finished" podID="dad38199-026b-4603-8647-08f578e29c6f" containerID="a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6" exitCode=60
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.715891    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerDied","Data":"a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6"}
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.716109    4550 scope.go:122] "RemoveContainer" containerID="a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6"
Nov 03 00:30:04.717512 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:04.716328    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw_odh-ai-gateway-infra(dad38199-026b-4603-8647-08f578e29c6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" podUID="dad38199-026b-4603-8647-08f578e29c6f"
Nov 03 00:30:04.719265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.719241    4550 generic.go:356] "Generic (PLEG): container finished" podID="d53ab800-74b9-4633-b863-c8129b2247f7" containerID="fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f" exitCode=60
Nov 03 00:30:04.719381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.719317    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerDied","Data":"fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f"}
Nov 03 00:30:04.719582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.719566    4550 scope.go:122] "RemoveContainer" containerID="fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f"
Nov 03 00:30:04.719754 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:04.719737    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px_odh-ai-gateway-infra(d53ab800-74b9-4633-b863-c8129b2247f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" podUID="d53ab800-74b9-4633-b863-c8129b2247f7"
Nov 03 00:30:04.721332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.721309    4550 generic.go:356] "Generic (PLEG): container finished" podID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerID="722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c" exitCode=60
Nov 03 00:30:04.721423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.721342    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerDied","Data":"722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c"}
Nov 03 00:30:04.721593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.721579    4550 scope.go:122] "RemoveContainer" containerID="722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c"
Nov 03 00:30:04.721762 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:04.721745    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg_odh-ai-gateway-infra(d26ea9ca-7b6a-4569-abe3-5523c98afd35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35"
Nov 03 00:30:04.725184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.725165    4550 scope.go:122] "RemoveContainer" containerID="1b57b31c0dd3d649c27394e6a990795bef92b1b1676860bfe58070d782bdd710"
Nov 03 00:30:04.781780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.753633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:04.893240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.892860    4550 scope.go:122] "RemoveContainer" containerID="23246472e01daeea67246dc043a9f166d5f2c16314a9588dd000637ada733e03"
Nov 03 00:30:04.958522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.958490    4550 scope.go:122] "RemoveContainer" containerID="5de94d12a81d7824328ab84e392f8512ab54b3bde5a3447c96b77754c5238584"
Nov 03 00:30:04.974605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.974571    4550 scope.go:122] "RemoveContainer" containerID="3bacb080be70c7ca0c8d0f131a7a74e69bedc976adec0085a6309cc56e03e3ac"
Nov 03 00:30:04.987700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.987663    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"]
Nov 03 00:30:04.994692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.994670    4550 scope.go:122] "RemoveContainer" containerID="f176ea02bbc0eb057fb1cb14e6a82402285f89008385222d5df84d3af085fbc2"
Nov 03 00:30:04.996913 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:30:04.996887    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1dcf41_93f4_4cfd_8a2c_f27465d0e45e.slice/crio-a5af8daf97aa68f174e4a27e5cb39fcbad6714f7c9a42221d02d0e0b6d7f983d WatchSource:0}: Error finding container a5af8daf97aa68f174e4a27e5cb39fcbad6714f7c9a42221d02d0e0b6d7f983d: Status 404 returned error can't find the container with id a5af8daf97aa68f174e4a27e5cb39fcbad6714f7c9a42221d02d0e0b6d7f983d
Nov 03 00:30:04.999598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.999568    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:04.999690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.999632    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:04.999690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:04.999662    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:05.737447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.735612    4550 scope.go:122] "RemoveContainer" containerID="d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0"
Nov 03 00:30:05.737447 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:05.735898    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk_odh-ai-gateway-infra(b1b691a4-0be8-4942-b5d4-d7236acb1cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7"
Nov 03 00:30:05.737447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.737163    4550 scope.go:122] "RemoveContainer" containerID="c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e"
Nov 03 00:30:05.737888 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:05.737464    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-cr6gl_odh-ai-gateway-infra(e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5"
Nov 03 00:30:05.740464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.738665    4550 scope.go:122] "RemoveContainer" containerID="98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e"
Nov 03 00:30:05.740464 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:05.738888    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6_odh-ai-gateway-infra(40e53f58-dbc7-4da5-9815-84620740bc54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" podUID="40e53f58-dbc7-4da5-9815-84620740bc54"
Nov 03 00:30:05.740773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.740757    4550 scope.go:122] "RemoveContainer" containerID="a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6"
Nov 03 00:30:05.740941 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:05.740924    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw_odh-ai-gateway-infra(dad38199-026b-4603-8647-08f578e29c6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" podUID="dad38199-026b-4603-8647-08f578e29c6f"
Nov 03 00:30:05.742815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.742795    4550 scope.go:122] "RemoveContainer" containerID="fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f"
Nov 03 00:30:05.742977 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:05.742958    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px_odh-ai-gateway-infra(d53ab800-74b9-4633-b863-c8129b2247f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" podUID="d53ab800-74b9-4633-b863-c8129b2247f7"
Nov 03 00:30:05.744964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.744936    4550 scope.go:122] "RemoveContainer" containerID="722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c"
Nov 03 00:30:05.745164 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:05.745141    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg_odh-ai-gateway-infra(d26ea9ca-7b6a-4569-abe3-5523c98afd35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35"
Nov 03 00:30:05.746554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.746519    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" event={"ID":"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e","Type":"ContainerStarted","Data":"b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"}
Nov 03 00:30:05.746649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.746581    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" event={"ID":"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e","Type":"ContainerStarted","Data":"a5af8daf97aa68f174e4a27e5cb39fcbad6714f7c9a42221d02d0e0b6d7f983d"}
Nov 03 00:30:05.865088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:05.865027    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podStartSLOduration=1.8650082719999999 podStartE2EDuration="1.865008272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:05.863566069 +0000 UTC m=+1488.185102898" watchObservedRunningTime="2026-11-03 00:30:05.865008272 +0000 UTC m=+1488.186545078"
Nov 03 00:30:06.553960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:06.553925    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:06.560265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:06.560239    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:06.745331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:06.745298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:06.754118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:06.754091    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:06.755195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:06.755172    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:30:06.844006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:06.843897    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:07.013653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:07.013625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:07.624644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:07.624612    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:30:08.258354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:08.258317    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:08.369664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:08.369628    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:08.984073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:08.984040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:10.028909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:10.028870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:10.129123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:10.129090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:10.348332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:10.324596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:11.047538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:11.047498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:13.303299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:13.303254    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:14.801096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:14.801066    4550 scope.go:122] "RemoveContainer" containerID="a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6"
Nov 03 00:30:14.925384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:14.925279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:15.018963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:15.018933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:15.123359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:15.122268    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:15.812482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:15.812414    4550 scope.go:122] "RemoveContainer" containerID="fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f"
Nov 03 00:30:15.838290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:15.838248    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerStarted","Data":"2f3a17638e97104ce615475d73374be0914a25dcc0ffc17dbaf2d186ef3735c2"}
Nov 03 00:30:16.800853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.800827    4550 scope.go:122] "RemoveContainer" containerID="c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e"
Nov 03 00:30:16.845512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.845486    4550 generic.go:356] "Generic (PLEG): container finished" podID="dad38199-026b-4603-8647-08f578e29c6f" containerID="2f3a17638e97104ce615475d73374be0914a25dcc0ffc17dbaf2d186ef3735c2" exitCode=60
Nov 03 00:30:16.845849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.845521    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerDied","Data":"2f3a17638e97104ce615475d73374be0914a25dcc0ffc17dbaf2d186ef3735c2"}
Nov 03 00:30:16.845849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.845579    4550 scope.go:122] "RemoveContainer" containerID="a39a13d2dfc6d9fdbe82a4a8ef6463f29b22b1256445fc4b4111bc3245e3d9f6"
Nov 03 00:30:16.845963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.845947    4550 scope.go:122] "RemoveContainer" containerID="2f3a17638e97104ce615475d73374be0914a25dcc0ffc17dbaf2d186ef3735c2"
Nov 03 00:30:16.846286 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:16.846259    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw_odh-ai-gateway-infra(dad38199-026b-4603-8647-08f578e29c6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" podUID="dad38199-026b-4603-8647-08f578e29c6f"
Nov 03 00:30:16.847724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.847705    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerStarted","Data":"cfb92a96874823047ad5424ca9706dd67246f14439740877accf03adc677a648"}
Nov 03 00:30:16.887886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:16.887859    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"]
Nov 03 00:30:17.058358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.058252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:17.808487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.808461    4550 scope.go:122] "RemoveContainer" containerID="722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c"
Nov 03 00:30:17.808670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.808605    4550 scope.go:122] "RemoveContainer" containerID="98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e"
Nov 03 00:30:17.855792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.855768    4550 generic.go:356] "Generic (PLEG): container finished" podID="d53ab800-74b9-4633-b863-c8129b2247f7" containerID="cfb92a96874823047ad5424ca9706dd67246f14439740877accf03adc677a648" exitCode=60
Nov 03 00:30:17.856157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.855830    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerDied","Data":"cfb92a96874823047ad5424ca9706dd67246f14439740877accf03adc677a648"}
Nov 03 00:30:17.856157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.855917    4550 scope.go:122] "RemoveContainer" containerID="fa07198b918e104280d4374fa4830f5089bdfe670df8be2038fc2caea930363f"
Nov 03 00:30:17.856257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.856213    4550 scope.go:122] "RemoveContainer" containerID="cfb92a96874823047ad5424ca9706dd67246f14439740877accf03adc677a648"
Nov 03 00:30:17.856511 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:17.856491    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px_odh-ai-gateway-infra(d53ab800-74b9-4633-b863-c8129b2247f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" podUID="d53ab800-74b9-4633-b863-c8129b2247f7"
Nov 03 00:30:17.858334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.858305    4550 generic.go:356] "Generic (PLEG): container finished" podID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerID="09e509c976a60708999ece381f730238ba4236266a1614c0a56e7fe791febb82" exitCode=60
Nov 03 00:30:17.858426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.858364    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerDied","Data":"09e509c976a60708999ece381f730238ba4236266a1614c0a56e7fe791febb82"}
Nov 03 00:30:17.858723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.858704    4550 scope.go:122] "RemoveContainer" containerID="09e509c976a60708999ece381f730238ba4236266a1614c0a56e7fe791febb82"
Nov 03 00:30:17.858958 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:17.858933    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894430-cr6gl_odh-ai-gateway-infra(e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5"
Nov 03 00:30:17.873298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.873278    4550 scope.go:122] "RemoveContainer" containerID="c37d984fd3e8dd10de58cf1cb591ed175f4f5b41c895cb1a586183b2b170153e"
Nov 03 00:30:17.955035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:17.954788    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"]
Nov 03 00:30:18.051344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.051316    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:18.136947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.136859    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca\") pod \"dad38199-026b-4603-8647-08f578e29c6f\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") "
Nov 03 00:30:18.137114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.137095    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c\" (UniqueName: \"kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c\") pod \"dad38199-026b-4603-8647-08f578e29c6f\" (UID: \"dad38199-026b-4603-8647-08f578e29c6f\") "
Nov 03 00:30:18.137203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.137177    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca" pod "dad38199-026b-4603-8647-08f578e29c6f" (UID: "dad38199-026b-4603-8647-08f578e29c6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:18.137715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.137694    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dad38199-026b-4603-8647-08f578e29c6f-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:18.139185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.139165    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c" pod "dad38199-026b-4603-8647-08f578e29c6f" (UID: "dad38199-026b-4603-8647-08f578e29c6f"). InnerVolumeSpecName "kube-api-access-8nr2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:18.231388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.231339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:18.239102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.239067    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nr2c\" (UniqueName: \"kubernetes.io/projected/dad38199-026b-4603-8647-08f578e29c6f-kube-api-access-8nr2c\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:18.342710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.342665    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:18.439723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.439690    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:18.564336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.564305    4550 scope.go:122] "RemoveContainer" containerID="2dc19b4c23af9deabaa0daaa75362c6cb51321b6bac84bc01c0e1e344ac5bb3d"
Nov 03 00:30:18.581623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.581598    4550 scope.go:122] "RemoveContainer" containerID="264209fb4f00f670f787389aa03856ab25d688544744cefb32ad2340f198f9c5"
Nov 03 00:30:18.592554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.592527    4550 scope.go:122] "RemoveContainer" containerID="bac6ec73ed079827afb2d33b805d4bda97949c33d2f9ef4eb6029cbb016d10ec"
Nov 03 00:30:18.801800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.801772    4550 scope.go:122] "RemoveContainer" containerID="d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0"
Nov 03 00:30:18.934475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.931888    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerStarted","Data":"7372c6f7ba5d519605c84056b7ae147b24631699c07abd6b140d19591b57056b"}
Nov 03 00:30:18.939375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.939338    4550 generic.go:356] "Generic (PLEG): container finished" podID="40e53f58-dbc7-4da5-9815-84620740bc54" containerID="9ee1efaace1432d6a66c9fbb5f89349e8d48de6ba8441210aabdccc1822ed085" exitCode=60
Nov 03 00:30:18.939515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.939478    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerDied","Data":"9ee1efaace1432d6a66c9fbb5f89349e8d48de6ba8441210aabdccc1822ed085"}
Nov 03 00:30:18.939593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.939550    4550 scope.go:122] "RemoveContainer" containerID="98888b1e48006c51ca6e98defc34cf19aa0696648d9946548c69fdfd8072e72e"
Nov 03 00:30:18.940125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.940102    4550 scope.go:122] "RemoveContainer" containerID="9ee1efaace1432d6a66c9fbb5f89349e8d48de6ba8441210aabdccc1822ed085"
Nov 03 00:30:18.940581 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:18.940482    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6_odh-ai-gateway-infra(40e53f58-dbc7-4da5-9815-84620740bc54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" podUID="40e53f58-dbc7-4da5-9815-84620740bc54"
Nov 03 00:30:18.964024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.944625    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw" event={"ID":"dad38199-026b-4603-8647-08f578e29c6f","Type":"ContainerDied","Data":"c4d9cd3858d236174a87f02626a4c0067c370777d8f2dcb9f0b6b2efb100a4ea"}
Nov 03 00:30:18.964024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.944721    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"
Nov 03 00:30:18.985665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.981033    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:19.000147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:18.999480    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"]
Nov 03 00:30:19.101566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.101539    4550 scope.go:122] "RemoveContainer" containerID="2f3a17638e97104ce615475d73374be0914a25dcc0ffc17dbaf2d186ef3735c2"
Nov 03 00:30:19.116462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.114281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:19.128733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.128702    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"]
Nov 03 00:30:19.137694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.137666    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8e90c5-29894430-fdcnw"]
Nov 03 00:30:19.229098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.229044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:19.287960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.287659    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:19.317033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.317010    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:19.375728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.371618    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7\" (UniqueName: \"kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7\") pod \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") "
Nov 03 00:30:19.375728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.371767    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr\" (UniqueName: \"kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr\") pod \"d53ab800-74b9-4633-b863-c8129b2247f7\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") "
Nov 03 00:30:19.375728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.371806    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca\") pod \"d53ab800-74b9-4633-b863-c8129b2247f7\" (UID: \"d53ab800-74b9-4633-b863-c8129b2247f7\") "
Nov 03 00:30:19.375728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.371868    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca\") pod \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\" (UID: \"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5\") "
Nov 03 00:30:19.375728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.373267    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca" pod "d53ab800-74b9-4633-b863-c8129b2247f7" (UID: "d53ab800-74b9-4633-b863-c8129b2247f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:19.375728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.373415    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca" pod "e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" (UID: "e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:19.376159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.375811    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr" pod "d53ab800-74b9-4633-b863-c8129b2247f7" (UID: "d53ab800-74b9-4633-b863-c8129b2247f7"). InnerVolumeSpecName "kube-api-access-tp7mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:19.376667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.376641    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7" pod "e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" (UID: "e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5"). InnerVolumeSpecName "kube-api-access-xw7p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:19.473921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.473886    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp7mr\" (UniqueName: \"kubernetes.io/projected/d53ab800-74b9-4633-b863-c8129b2247f7-kube-api-access-tp7mr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:19.473921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.473922    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d53ab800-74b9-4633-b863-c8129b2247f7-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:19.474109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.473938    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:19.474109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.473952    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw7p7\" (UniqueName: \"kubernetes.io/projected/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5-kube-api-access-xw7p7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:19.846673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.839707    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad38199-026b-4603-8647-08f578e29c6f" path="/var/lib/kubelet/pods/dad38199-026b-4603-8647-08f578e29c6f/volumes"
Nov 03 00:30:19.959184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.959147    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px" event={"ID":"d53ab800-74b9-4633-b863-c8129b2247f7","Type":"ContainerDied","Data":"f9ba54320d3b13e41aa6115800b23b77dc3424529f80e851180398c81d82bae5"}
Nov 03 00:30:19.959839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.959209    4550 scope.go:122] "RemoveContainer" containerID="cfb92a96874823047ad5424ca9706dd67246f14439740877accf03adc677a648"
Nov 03 00:30:19.959839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.959316    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"
Nov 03 00:30:19.963255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.963227    4550 generic.go:356] "Generic (PLEG): container finished" podID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerID="7372c6f7ba5d519605c84056b7ae147b24631699c07abd6b140d19591b57056b" exitCode=60
Nov 03 00:30:19.963390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.963351    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerDied","Data":"7372c6f7ba5d519605c84056b7ae147b24631699c07abd6b140d19591b57056b"}
Nov 03 00:30:19.963764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.963740    4550 scope.go:122] "RemoveContainer" containerID="7372c6f7ba5d519605c84056b7ae147b24631699c07abd6b140d19591b57056b"
Nov 03 00:30:19.964001 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:19.963983    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg_odh-ai-gateway-infra(d26ea9ca-7b6a-4569-abe3-5523c98afd35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35"
Nov 03 00:30:19.966589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.966567    4550 generic.go:356] "Generic (PLEG): container finished" podID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerID="c5c8dd08d8306dd9759ad9297d90c1b6d6293a86558db4baca7d626c7f3b1363" exitCode=60
Nov 03 00:30:19.966714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.966630    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerDied","Data":"c5c8dd08d8306dd9759ad9297d90c1b6d6293a86558db4baca7d626c7f3b1363"}
Nov 03 00:30:19.967032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.967010    4550 scope.go:122] "RemoveContainer" containerID="c5c8dd08d8306dd9759ad9297d90c1b6d6293a86558db4baca7d626c7f3b1363"
Nov 03 00:30:19.967324 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:19.967292    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk_odh-ai-gateway-infra(b1b691a4-0be8-4942-b5d4-d7236acb1cb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7"
Nov 03 00:30:19.968821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.968799    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl" event={"ID":"e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5","Type":"ContainerDied","Data":"0a7eaaf70ecc2039ba60facf652bb62105ae4dd535c81c410d3a37eb2368d48b"}
Nov 03 00:30:19.969023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.968839    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"
Nov 03 00:30:19.972594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.972505    4550 scope.go:122] "RemoveContainer" containerID="722c3de5312dff4f123ba6bb9912169996acc9833efc706e543fcf421a84a80c"
Nov 03 00:30:19.997423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:19.997387    4550 scope.go:122] "RemoveContainer" containerID="d5752f057b48a9657ee14eab10f3587124b49988c396558ce1b4fde9eb07ddc0"
Nov 03 00:30:20.003358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.003324    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"]
Nov 03 00:30:20.013980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.013957    4550 scope.go:122] "RemoveContainer" containerID="09e509c976a60708999ece381f730238ba4236266a1614c0a56e7fe791febb82"
Nov 03 00:30:20.022131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.022103    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"]
Nov 03 00:30:20.039218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.039187    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-2af7b0-29894430-wk5px"]
Nov 03 00:30:20.068986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.068959    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"]
Nov 03 00:30:20.116606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.116521    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"]
Nov 03 00:30:20.119724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.119688    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894430-cr6gl"]
Nov 03 00:30:20.270985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.270954    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:20.400809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.400738    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5\" (UniqueName: \"kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5\") pod \"40e53f58-dbc7-4da5-9815-84620740bc54\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") "
Nov 03 00:30:20.401001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.400927    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca\") pod \"40e53f58-dbc7-4da5-9815-84620740bc54\" (UID: \"40e53f58-dbc7-4da5-9815-84620740bc54\") "
Nov 03 00:30:20.401976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.401947    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca" pod "40e53f58-dbc7-4da5-9815-84620740bc54" (UID: "40e53f58-dbc7-4da5-9815-84620740bc54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:20.409005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.408975    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5" pod "40e53f58-dbc7-4da5-9815-84620740bc54" (UID: "40e53f58-dbc7-4da5-9815-84620740bc54"). InnerVolumeSpecName "kube-api-access-g9vz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:20.503837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.503797    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40e53f58-dbc7-4da5-9815-84620740bc54-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.503837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.503838    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vz5\" (UniqueName: \"kubernetes.io/projected/40e53f58-dbc7-4da5-9815-84620740bc54-kube-api-access-g9vz5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:20.982058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.982029    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"
Nov 03 00:30:20.982058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.982044    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6" event={"ID":"40e53f58-dbc7-4da5-9815-84620740bc54","Type":"ContainerDied","Data":"1ab3238895d7ebd104012543d5ff35e98826920c06cb21644dd282263ce1cc89"}
Nov 03 00:30:20.982612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:20.982097    4550 scope.go:122] "RemoveContainer" containerID="9ee1efaace1432d6a66c9fbb5f89349e8d48de6ba8441210aabdccc1822ed085"
Nov 03 00:30:21.084056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.083983    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"]
Nov 03 00:30:21.092592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.092559    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894430-4rrt6"]
Nov 03 00:30:21.095693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.095674    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"]
Nov 03 00:30:21.172903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.172872    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:21.222246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.222089    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4\" (UniqueName: \"kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4\") pod \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") "
Nov 03 00:30:21.222246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.222165    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca\") pod \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\" (UID: \"d26ea9ca-7b6a-4569-abe3-5523c98afd35\") "
Nov 03 00:30:21.222970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.222941    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca" pod "d26ea9ca-7b6a-4569-abe3-5523c98afd35" (UID: "d26ea9ca-7b6a-4569-abe3-5523c98afd35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:21.237645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.237595    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4" pod "d26ea9ca-7b6a-4569-abe3-5523c98afd35" (UID: "d26ea9ca-7b6a-4569-abe3-5523c98afd35"). InnerVolumeSpecName "kube-api-access-89lb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:21.315108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.315085    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:21.323736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.323709    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lb4\" (UniqueName: \"kubernetes.io/projected/d26ea9ca-7b6a-4569-abe3-5523c98afd35-kube-api-access-89lb4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.323736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.323736    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d26ea9ca-7b6a-4569-abe3-5523c98afd35-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.425202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.425170    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs\" (UniqueName: \"kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs\") pod \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") "
Nov 03 00:30:21.425368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.425270    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca\") pod \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\" (UID: \"b1b691a4-0be8-4942-b5d4-d7236acb1cb7\") "
Nov 03 00:30:21.425883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.425857    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca" pod "b1b691a4-0be8-4942-b5d4-d7236acb1cb7" (UID: "b1b691a4-0be8-4942-b5d4-d7236acb1cb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:21.428336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.428308    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs" pod "b1b691a4-0be8-4942-b5d4-d7236acb1cb7" (UID: "b1b691a4-0be8-4942-b5d4-d7236acb1cb7"). InnerVolumeSpecName "kube-api-access-98lgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:21.474215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.474186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:21.528529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.528448    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98lgs\" (UniqueName: \"kubernetes.io/projected/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-kube-api-access-98lgs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.528529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.528484    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b691a4-0be8-4942-b5d4-d7236acb1cb7-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:30:21.808243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.808169    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" path="/var/lib/kubelet/pods/40e53f58-dbc7-4da5-9815-84620740bc54/volumes"
Nov 03 00:30:21.808628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.808614    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" path="/var/lib/kubelet/pods/d53ab800-74b9-4633-b863-c8129b2247f7/volumes"
Nov 03 00:30:21.808979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.808967    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" path="/var/lib/kubelet/pods/e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5/volumes"
Nov 03 00:30:21.988731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.988698    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"
Nov 03 00:30:21.988731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.988712    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg" event={"ID":"d26ea9ca-7b6a-4569-abe3-5523c98afd35","Type":"ContainerDied","Data":"7ff2c192e561c5c6f3bc4af048d3d66df14ad2380f084086f2b2600353511376"}
Nov 03 00:30:21.989279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.988797    4550 scope.go:122] "RemoveContainer" containerID="7372c6f7ba5d519605c84056b7ae147b24631699c07abd6b140d19591b57056b"
Nov 03 00:30:21.990141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.990128    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"
Nov 03 00:30:21.990238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.990168    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk" event={"ID":"b1b691a4-0be8-4942-b5d4-d7236acb1cb7","Type":"ContainerDied","Data":"3b22860aa66119f535d1658dfaeac0578ae2123fd265c568202a55cd154f4291"}
Nov 03 00:30:21.998636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:21.998556    4550 scope.go:122] "RemoveContainer" containerID="c5c8dd08d8306dd9759ad9297d90c1b6d6293a86558db4baca7d626c7f3b1363"
Nov 03 00:30:22.046502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:22.046471    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"]
Nov 03 00:30:22.049844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:22.049819    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894430-8qznk"]
Nov 03 00:30:22.084479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:22.084391    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"]
Nov 03 00:30:22.090005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:22.089981    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894430-bz9lg"]
Nov 03 00:30:23.808964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:23.808931    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" path="/var/lib/kubelet/pods/b1b691a4-0be8-4942-b5d4-d7236acb1cb7/volumes"
Nov 03 00:30:23.809366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:23.809352    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" path="/var/lib/kubelet/pods/d26ea9ca-7b6a-4569-abe3-5523c98afd35/volumes"
Nov 03 00:30:23.845911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:23.845876    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:24.723726 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:24.723686    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:30:25.510318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:25.510283    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:27.288120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:27.288078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:29.025825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:29.025787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:30.774820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:30.774784    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:32.575827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:32.575790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:34.727857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:34.727823    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:36.524837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:36.524800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:36.643866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:36.643822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:36.924853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:36.924823    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:37.231974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:37.231899    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:38.417516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:38.417475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:40.182016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:40.181981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:40.277921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:40.277884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:42.881934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:42.881895    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:44.261389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:44.261345    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:44.372719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:44.368931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:45.642502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:45.642464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:47.493274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:47.493234    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:49.277804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:49.277766    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:51.159885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:51.159846    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:52.041910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:52.041872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:53.551309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:53.551271    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:54.722931 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:30:54.722891    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:30:54.885550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:54.885510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:55.373490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:55.373452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:55.988048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:55.988008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:56.376878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.376643    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"]
Nov 03 00:30:56.377827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.377786    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:30:56.377999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.377985    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:30:56.378086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378077    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:30:56.378158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378149    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:30:56.378233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378223    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:30:56.378322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378303    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:30:56.378421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378412    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:30:56.378514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378504    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:30:56.378607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378597    4550 cpu_manager.go:414] "containerMap: removing container" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:30:56.378762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378738    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:30:56.378856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378837    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:30:56.378856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378851    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378877    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378885    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378905    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378912    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378923    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378930    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378942    4550 cpu_manager.go:414] "containerMap: removing container" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378949    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378959    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.378966    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379011    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:30:56.379054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379017    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379143    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379153    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379163    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379172    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379180    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379189    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379199    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379207    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379218    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379228    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379236    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379244    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379252    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:30:56.379616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.379260    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:30:56.383732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.383682    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.417583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.402157    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-68738d-openshift-default-dockercfg-q5v4s\""
Nov 03 00:30:56.419114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.418924    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"]
Nov 03 00:30:56.435850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.435827    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:56.565456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.539698    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.593950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.593916    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bs4\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.594281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.594258    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.599579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.599528    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.599837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.599819    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.607676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.607649    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.607934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.607916    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.608099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.608084    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.608211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.608198    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.719917    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.719980    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720066    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720113    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720196    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720279    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720305    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720372    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.723454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.720400    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bs4\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.727451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.725153    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.727451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.725887    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.727451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.726128    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.741155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.740008    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.741155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.740277    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.752479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.741906    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.789053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.789019    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.792102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.792043    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.799107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.799079    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bs4\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4\") pod \"e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") " pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:56.825392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:56.825356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:57.020731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:57.020638    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:57.376508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:57.375984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:57.395454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:57.378332    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"]
Nov 03 00:30:57.395454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:57.388388    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:57.395454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:57.388489    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:57.395454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:57.388558    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:30:58.253101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:58.253071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:58.272704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:58.256835    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" event={"ID":"58c940e7-9d85-4bf7-8a9d-45520f06a151","Type":"ContainerStarted","Data":"16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"}
Nov 03 00:30:58.272704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:58.256910    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" event={"ID":"58c940e7-9d85-4bf7-8a9d-45520f06a151","Type":"ContainerStarted","Data":"11fb21bb76769cd311e3acfac033d74a49049c27b9f82a234f24c90eafd4ed36"}
Nov 03 00:30:58.317466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:58.317378    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podStartSLOduration=2.317360849 podStartE2EDuration="2.317360849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:58.316781176 +0000 UTC m=+1540.638317975" watchObservedRunningTime="2026-11-03 00:30:58.317360849 +0000 UTC m=+1540.638897649"
Nov 03 00:30:59.020841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:59.020803    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:59.027771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:59.027731    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:59.151636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:59.151606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:30:59.299511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:59.296822    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:30:59.309456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:30:59.304599    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:31:00.781721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:00.768368    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:00.885828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:00.885793    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:00.988233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:00.988142    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:02.486596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:02.486565    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:02.588132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:02.588093    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:03.099234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:03.099197    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:05.411141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:05.411087    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:06.332073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.332041    4550 generic.go:356] "Generic (PLEG): container finished" podID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerID="b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99" exitCode=0
Nov 03 00:31:06.332331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.332118    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" event={"ID":"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e","Type":"ContainerDied","Data":"b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"}
Nov 03 00:31:06.332643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.332624    4550 scope.go:122] "RemoveContainer" containerID="b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"
Nov 03 00:31:06.334092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.334063    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:31:06.334164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.334134    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:31:06.334202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.334166    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:31:06.754498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.754466    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:31:06.766828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.766794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:06.951460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:06.950484    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:07.345898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.345864    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" event={"ID":"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e","Type":"ContainerStarted","Data":"6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7"}
Nov 03 00:31:07.347795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.347771    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"]
Nov 03 00:31:07.348291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348275    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:31:07.348368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348293    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53ab800-74b9-4633-b863-c8129b2247f7" containerName="cleanup"
Nov 03 00:31:07.348368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348311    4550 cpu_manager.go:414] "containerMap: removing container" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:31:07.348368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348318    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:31:07.348368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348328    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:31:07.348368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348336    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26ea9ca-7b6a-4569-abe3-5523c98afd35" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348382    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348390    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348400    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348408    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348443    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348451    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348636    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9eeace3-0a7d-4f8b-8083-80ae7d80e6f5" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348651    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad38199-026b-4603-8647-08f578e29c6f" containerName="cleanup"
Nov 03 00:31:07.348661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348662    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1b691a4-0be8-4942-b5d4-d7236acb1cb7" containerName="cleanup"
Nov 03 00:31:07.349094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.348670    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e53f58-dbc7-4da5-9815-84620740bc54" containerName="cleanup"
Nov 03 00:31:07.351863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.351844    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:07.380422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.373801    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:07.380422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.373933    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9xk\" (UniqueName: \"kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk\") pod \"payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:07.446781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.444912    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ea275f-dockercfg-dbqrk\""
Nov 03 00:31:07.446781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.445057    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea275f\""
Nov 03 00:31:07.451368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.451333    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"]
Nov 03 00:31:07.516570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.511455    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:07.533940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.524612    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9xk\" (UniqueName: \"kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk\") pod \"payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:07.542569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.542531    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:07.619727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.608508    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:31:07.693911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.693872    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:31:07.694118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.693950    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 00:31:07.742460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:07.742196    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9xk\" (UniqueName: \"kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk\") pod \"payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:08.040684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:08.040653    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:08.281456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:08.255584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:08.410644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:08.410592    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"]
Nov 03 00:31:08.554587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:08.554556    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:31:08.554754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:08.554607    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 00:31:09.387457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:09.377868    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5" event={"ID":"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7","Type":"ContainerStarted","Data":"c50daafc641ac8bc5fcfead56f2f22e660cb65a1a752a51f3fe7bf1f86a18ad1"}
Nov 03 00:31:09.387457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:09.377928    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5" event={"ID":"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7","Type":"ContainerStarted","Data":"e3edd2d9adb858632ac6838e30b854876d4afbe72a4608184afb0aa27fa9b69b"}
Nov 03 00:31:09.387457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:09.377973    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:09.421366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:09.421297    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5" podStartSLOduration=2.421276861 podStartE2EDuration="2.421276861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:09.420187565 +0000 UTC m=+1551.741724367" watchObservedRunningTime="2026-11-03 00:31:09.421276861 +0000 UTC m=+1551.742813664"
Nov 03 00:31:09.554997 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:09.554961    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:31:09.555177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:09.555024    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 00:31:10.221479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:10.221415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:10.383510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:10.376309    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:10.554222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:10.554187    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:31:10.554655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:10.554246    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 00:31:11.554225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:11.554194    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:31:11.554678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:11.554247    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 00:31:12.081450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:12.081050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:12.553906 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:12.553864    4550 patch_prober.go:29] interesting pod/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:31:12.554093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:12.553937    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 00:31:13.559329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:13.559298    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:31:13.559793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:13.559716    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:31:13.560523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:13.560505    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:31:14.366675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:14.366641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:14.709216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:14.709185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:14.806783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:14.806592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:14.910701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:14.910667    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:16.723280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:16.723247    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:18.537793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:18.537754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:20.231979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:20.231944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:20.842085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:20.842045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:21.396020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:21.395989    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:31:22.088345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:22.088296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:23.205996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:23.205961    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:24.320856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:24.320821    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:24.723750 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:31:24.723713    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:31:26.003471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:26.003425    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:27.634024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:27.633983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:29.441412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:29.441332    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:29.552390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:29.552352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:31.186530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:31.186491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:31.282594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:31.282550    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:31.383774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:31.383737    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:32.953619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:32.953586    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:34.355265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:34.355237    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:34.617146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:34.617067    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:34.716609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:34.716576    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:35.266159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:35.266123    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:37.760847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:37.760811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:38.992190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:38.992152    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:39.174455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:39.151285    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:40.235267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:40.235234    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:42.041733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:42.041682    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:44.254559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:44.254516    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:44.357087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:44.357050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:45.041651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:45.041620    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:45.426343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:45.426310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:46.813699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:46.813669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:48.425725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:48.425689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:50.227701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:50.227664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:51.267868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:51.267833    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:52.031160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:52.031128    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:54.196577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:54.196533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:54.723486 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:31:54.723453    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:31:55.907649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:55.907590    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:57.563890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:57.563846    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:57.664692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:57.664650    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:57.776078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:57.776040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:31:59.530941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:31:59.530904    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:00.338691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.338645    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:32:00.338881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.338729    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:32:00.794648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.794556    4550 generic.go:356] "Generic (PLEG): container finished" podID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerID="16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08" exitCode=0
Nov 03 00:32:00.794648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.794632    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" event={"ID":"58c940e7-9d85-4bf7-8a9d-45520f06a151","Type":"ContainerDied","Data":"16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"}
Nov 03 00:32:00.795241 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.795101    4550 scope.go:122] "RemoveContainer" containerID="16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"
Nov 03 00:32:00.796755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.796728    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:32:00.796865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.796796    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:32:00.796865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:00.796827    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:32:01.204453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:01.204407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:01.809183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:01.809146    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" event={"ID":"58c940e7-9d85-4bf7-8a9d-45520f06a151","Type":"ContainerStarted","Data":"654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225"}
Nov 03 00:32:02.020808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:02.020776    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:32:02.020980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:02.020911    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 00:32:02.020980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:02.020959    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 00:32:02.985456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:02.980126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:03.025481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:03.021401    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 00:32:03.025481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:03.021475    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 00:32:04.021914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.021875    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 00:32:04.022457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.021945    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 00:32:04.194336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.194304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:04.432768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.432736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:04.545741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.545710    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"]
Nov 03 00:32:04.545998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.545974    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5" podUID="58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" containerName="payload-processing" containerID="cri-o://c50daafc641ac8bc5fcfead56f2f22e660cb65a1a752a51f3fe7bf1f86a18ad1" gracePeriod=30
Nov 03 00:32:04.655049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.655012    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:04.750359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.750322    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:04.847496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.847461    4550 generic.go:356] "Generic (PLEG): container finished" podID="58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" containerID="c50daafc641ac8bc5fcfead56f2f22e660cb65a1a752a51f3fe7bf1f86a18ad1" exitCode=0
Nov 03 00:32:04.847870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:04.847846    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5" event={"ID":"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7","Type":"ContainerDied","Data":"c50daafc641ac8bc5fcfead56f2f22e660cb65a1a752a51f3fe7bf1f86a18ad1"}
Nov 03 00:32:05.038531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.029526    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 00:32:05.038531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.029588    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 00:32:05.239074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.239036    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"]
Nov 03 00:32:05.239307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.239288    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy" containerID="cri-o://91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60" gracePeriod=30
Nov 03 00:32:05.349107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.349069    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"]
Nov 03 00:32:05.350293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.350224    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:32:05.352554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.352512    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy" containerID="cri-o://4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c" gracePeriod=30
Nov 03 00:32:05.416621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.416586    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"]
Nov 03 00:32:05.417161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.417123    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy" containerID="cri-o://972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85" gracePeriod=30
Nov 03 00:32:05.436377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.435414    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk\" (UniqueName: \"kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk\") pod \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") "
Nov 03 00:32:05.436377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.435702    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume\") pod \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\" (UID: \"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7\") "
Nov 03 00:32:05.436933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.436893    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume" pod "58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" (UID: "58fb88e0-3d7c-4963-a65c-7e8ab4e378e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:05.444522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.442921    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk" pod "58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" (UID: "58fb88e0-3d7c-4963-a65c-7e8ab4e378e7"). InnerVolumeSpecName "kube-api-access-4b9xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:05.536896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.536866    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:05.536896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.536894    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9xk\" (UniqueName: \"kubernetes.io/projected/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7-kube-api-access-4b9xk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:05.547071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.547046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:05.855011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.854982    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"
Nov 03 00:32:05.855183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.855024    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5" event={"ID":"58fb88e0-3d7c-4963-a65c-7e8ab4e378e7","Type":"ContainerDied","Data":"e3edd2d9adb858632ac6838e30b854876d4afbe72a4608184afb0aa27fa9b69b"}
Nov 03 00:32:05.855183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.855086    4550 scope.go:122] "RemoveContainer" containerID="c50daafc641ac8bc5fcfead56f2f22e660cb65a1a752a51f3fe7bf1f86a18ad1"
Nov 03 00:32:05.924775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.924739    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"]
Nov 03 00:32:05.928358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:05.928333    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-7d6c7ddb56-ccjl5"]
Nov 03 00:32:06.021705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:06.021666    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 00:32:06.021897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:06.021743    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 00:32:07.021689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:07.021654    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 03 00:32:07.022111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:07.021727    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 03 00:32:07.046845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:07.046811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:07.820192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:07.820154    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" path="/var/lib/kubelet/pods/58fb88e0-3d7c-4963-a65c-7e8ab4e378e7/volumes"
Nov 03 00:32:08.026896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:08.026865    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:32:08.027358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:08.027167    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:32:08.027809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:08.027789    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:32:08.030004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:08.029967    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:09.106717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:09.106678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:09.577480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:09.576634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:09.757673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:09.756256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:09.854199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:09.854114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:10.492581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.492549    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:10.593654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.593625    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:32:10.599074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.598994    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:10.757833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.757806    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.757986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.757873    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.757986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.757942    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.757992    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758041    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758094    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.758210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758180    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758211    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758224    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.758382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758265    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758302    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data\") pod \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\" (UID: \"5c67dd83-f1c1-4a6d-b901-aae9800b7abc\") "
Nov 03 00:32:10.758975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758800    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.758975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758880    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.758975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.758899    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.759173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.759039    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.759173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.759096    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.760780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.760754    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.761025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.760985    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.761199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.761177    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "kube-api-access-v6qcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.761286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.761189    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy" pod "5c67dd83-f1c1-4a6d-b901-aae9800b7abc" (UID: "5c67dd83-f1c1-4a6d-b901-aae9800b7abc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.773425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.773401    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:32:10.775522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.775502    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:32:10.860004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.859969    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860017    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860072    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860168    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860195    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860234    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860291    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860320    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860350    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860380    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860495    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860525    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860562    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860595    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860637    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860665    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo\") pod \"0e5815f0-9438-448f-b116-35aaefb1c8b0\" (UID: \"0e5815f0-9438-448f-b116-35aaefb1c8b0\") "
Nov 03 00:32:10.860971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860708    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.860971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.860742    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo\") pod \"a1add410-cf1d-4bde-8548-452e7d7f6af4\" (UID: \"a1add410-cf1d-4bde-8548-452e7d7f6af4\") "
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861447    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861472    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861486    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861500    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861513    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861526    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.863327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.861541    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6qcw\" (UniqueName: \"kubernetes.io/projected/5c67dd83-f1c1-4a6d-b901-aae9800b7abc-kube-api-access-v6qcw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.864339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.863922    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.864339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.864238    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.864524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.864492    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.864686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.864660    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.865487    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.866053    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.866418    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.866642    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.866672    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.866998    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.867338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.867277    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.867855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.867827    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.870207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.870166    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "kube-api-access-4r7hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.871022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.870996    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "kube-api-access-5x9gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.871259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.871018    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.871259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.871144    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token" pod "0e5815f0-9438-448f-b116-35aaefb1c8b0" (UID: "0e5815f0-9438-448f-b116-35aaefb1c8b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.872290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.872271    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.874918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.874893    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy" pod "a1add410-cf1d-4bde-8548-452e7d7f6af4" (UID: "a1add410-cf1d-4bde-8548-452e7d7f6af4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.917644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.917546    4550 generic.go:356] "Generic (PLEG): container finished" podID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerID="4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c" exitCode=0
Nov 03 00:32:10.917644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.917613    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"
Nov 03 00:32:10.917898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.917687    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" event={"ID":"0e5815f0-9438-448f-b116-35aaefb1c8b0","Type":"ContainerDied","Data":"4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c"}
Nov 03 00:32:10.917898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.917730    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs" event={"ID":"0e5815f0-9438-448f-b116-35aaefb1c8b0","Type":"ContainerDied","Data":"349220541f5c4288779431b25e38d125680b482972f30f6eebdaeb49fb648083"}
Nov 03 00:32:10.917898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.917758    4550 scope.go:122] "RemoveContainer" containerID="4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c"
Nov 03 00:32:10.920272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.919952    4550 generic.go:356] "Generic (PLEG): container finished" podID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerID="91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60" exitCode=0
Nov 03 00:32:10.920272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.920009    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"
Nov 03 00:32:10.920272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.920001    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" event={"ID":"5c67dd83-f1c1-4a6d-b901-aae9800b7abc","Type":"ContainerDied","Data":"91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60"}
Nov 03 00:32:10.920272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.920200    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7" event={"ID":"5c67dd83-f1c1-4a6d-b901-aae9800b7abc","Type":"ContainerDied","Data":"f24acac1b7617ba09e5538526cfd0d0f9fd77edd4000fdbf365450c3b8c50352"}
Nov 03 00:32:10.922786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.922041    4550 generic.go:356] "Generic (PLEG): container finished" podID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerID="972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85" exitCode=0
Nov 03 00:32:10.922786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.922070    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" event={"ID":"a1add410-cf1d-4bde-8548-452e7d7f6af4","Type":"ContainerDied","Data":"972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85"}
Nov 03 00:32:10.922786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.922106    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx" event={"ID":"a1add410-cf1d-4bde-8548-452e7d7f6af4","Type":"ContainerDied","Data":"0a5550304627fa3168223d1d8ff982fbce4bb41f2996d0c32c17edd509f16ce5"}
Nov 03 00:32:10.922786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.922183    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"
Nov 03 00:32:10.932952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.932928    4550 scope.go:122] "RemoveContainer" containerID="4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c"
Nov 03 00:32:10.933374 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:10.933292    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c\": container with ID starting with 4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c not found: ID does not exist" containerID="4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c"
Nov 03 00:32:10.933374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.933333    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c"} err="failed to get container status \"4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c\": rpc error: code = NotFound desc = could not find container \"4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c\": container with ID starting with 4863531da6c2dd56922ed31e7dfbf008256a3644f02c091f8615ae9a4299d20c not found: ID does not exist"
Nov 03 00:32:10.933374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.933358    4550 scope.go:122] "RemoveContainer" containerID="91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60"
Nov 03 00:32:10.942962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.942940    4550 scope.go:122] "RemoveContainer" containerID="91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60"
Nov 03 00:32:10.943275 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:10.943258    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60\": container with ID starting with 91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60 not found: ID does not exist" containerID="91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60"
Nov 03 00:32:10.943364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.943286    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60"} err="failed to get container status \"91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60\": rpc error: code = NotFound desc = could not find container \"91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60\": container with ID starting with 91e68acd45686e8b50d93148447541c8e5ccf31d1f63107aee852e088a345e60 not found: ID does not exist"
Nov 03 00:32:10.943364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.943310    4550 scope.go:122] "RemoveContainer" containerID="972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85"
Nov 03 00:32:10.953627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.953605    4550 scope.go:122] "RemoveContainer" containerID="972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85"
Nov 03 00:32:10.953951 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:10.953931    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85\": container with ID starting with 972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85 not found: ID does not exist" containerID="972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85"
Nov 03 00:32:10.953995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.953961    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85"} err="failed to get container status \"972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85\": rpc error: code = NotFound desc = could not find container \"972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85\": container with ID starting with 972c92d81688d60f3c01fa4e97dc5d4c7ee4b05fb9491fbb4d7f935777498b85 not found: ID does not exist"
Nov 03 00:32:10.962729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962703    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962729    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962738    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962748    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962757    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962766    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962774    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962783    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962791    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962798    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962806    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r7hf\" (UniqueName: \"kubernetes.io/projected/0e5815f0-9438-448f-b116-35aaefb1c8b0-kube-api-access-4r7hf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962814    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e5815f0-9438-448f-b116-35aaefb1c8b0-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962821    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962829    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1add410-cf1d-4bde-8548-452e7d7f6af4-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962838    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1add410-cf1d-4bde-8548-452e7d7f6af4-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962846    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962854    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x9gp\" (UniqueName: \"kubernetes.io/projected/a1add410-cf1d-4bde-8548-452e7d7f6af4-kube-api-access-5x9gp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.962876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:10.962862    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e5815f0-9438-448f-b116-35aaefb1c8b0-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.007097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.007057    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"]
Nov 03 00:32:11.009831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.009803    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-69bj7"]
Nov 03 00:32:11.045752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.045713    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"]
Nov 03 00:32:11.051678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.051650    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8pmlx"]
Nov 03 00:32:11.087855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.087824    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"]
Nov 03 00:32:11.090797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.090767    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-rz5fs"]
Nov 03 00:32:11.807479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.807442    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" path="/var/lib/kubelet/pods/0e5815f0-9438-448f-b116-35aaefb1c8b0/volumes"
Nov 03 00:32:11.808123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.808102    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" path="/var/lib/kubelet/pods/5c67dd83-f1c1-4a6d-b901-aae9800b7abc/volumes"
Nov 03 00:32:11.808639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:11.808618    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" path="/var/lib/kubelet/pods/a1add410-cf1d-4bde-8548-452e7d7f6af4/volumes"
Nov 03 00:32:12.419872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:12.419829    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:12.731620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:12.731530    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:12.833612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:12.833574    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:12.933038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:12.933002    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:13.525754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:13.525715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:13.915307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:13.915276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:14.630381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:14.630348    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:14.983086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:14.983015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:15.083365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:15.083331    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:15.600729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:15.600696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:16.536244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:16.536211    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:17.459283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:17.459237    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:17.560403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:17.560367    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:18.728955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:18.728924    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:18.829904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:18.829868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:19.446317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:19.446277    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:19.549315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:19.549276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:19.647593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:19.647563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:19.946240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:19.946205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:20.036461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:20.035713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:20.749422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:20.749381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:21.593946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:21.593910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:21.696528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:21.696496    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:21.799651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:21.799597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:22.363919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:22.363882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:22.742353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:22.742316    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:23.538532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.538497    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:23.696693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.696613    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:23.902125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902097    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"]
Nov 03 00:32:23.902642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902625    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy"
Nov 03 00:32:23.902701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902643    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy"
Nov 03 00:32:23.902701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902663    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy"
Nov 03 00:32:23.902701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902669    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy"
Nov 03 00:32:23.902701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902688    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy"
Nov 03 00:32:23.902701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902693    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy"
Nov 03 00:32:23.902940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902706    4550 cpu_manager.go:414] "containerMap: removing container" podUID="58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" containerName="payload-processing"
Nov 03 00:32:23.902940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902714    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" containerName="payload-processing"
Nov 03 00:32:23.902940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902790    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1add410-cf1d-4bde-8548-452e7d7f6af4" containerName="istio-proxy"
Nov 03 00:32:23.902940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902804    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e5815f0-9438-448f-b116-35aaefb1c8b0" containerName="istio-proxy"
Nov 03 00:32:23.902940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902812    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="58fb88e0-3d7c-4963-a65c-7e8ab4e378e7" containerName="payload-processing"
Nov 03 00:32:23.902940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.902818    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c67dd83-f1c1-4a6d-b901-aae9800b7abc" containerName="istio-proxy"
Nov 03 00:32:23.908305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.908278    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:23.924811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.924785    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"]
Nov 03 00:32:23.934128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.934107    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:32:23.934251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:23.934155    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:32:24.010151 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.010106    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.010367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.010260    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.010499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.010402    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5kkr\" (UniqueName: \"kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.112124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.112095    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.112307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.112213    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5kkr\" (UniqueName: \"kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.112368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.112331    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.112967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.112943    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.114521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.114503    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.132405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.132385    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5kkr\" (UniqueName: \"kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr\") pod \"maas-api-revoke-sub-7648e554-f98jp\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.222772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.222696    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:24.526118 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:32:24.526088    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24fad59_6017_48f0_8d43_8056174747a1.slice/crio-da03589b406ac4f6e331638e29174cc608e276f7fe93d9beb25df193a43b0bd8 WatchSource:0}: Error finding container da03589b406ac4f6e331638e29174cc608e276f7fe93d9beb25df193a43b0bd8: Status 404 returned error can't find the container with id da03589b406ac4f6e331638e29174cc608e276f7fe93d9beb25df193a43b0bd8
Nov 03 00:32:24.527309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.527286    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"]
Nov 03 00:32:24.723890 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:24.723850    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:32:24.782646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.782578    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:24.887143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:24.887112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:25.092520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:25.092415    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp" event={"ID":"a24fad59-6017-48f0-8d43-8056174747a1","Type":"ContainerStarted","Data":"bafb85bda8fdeab7b0732083c9b37c58d48ff8a349f535430714ff3115743a16"}
Nov 03 00:32:25.092520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:25.092499    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp" event={"ID":"a24fad59-6017-48f0-8d43-8056174747a1","Type":"ContainerStarted","Data":"da03589b406ac4f6e331638e29174cc608e276f7fe93d9beb25df193a43b0bd8"}
Nov 03 00:32:25.134155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:25.134092    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp" podStartSLOduration=2.134079425 podStartE2EDuration="2.134079425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:25.132520235 +0000 UTC m=+1627.454057025" watchObservedRunningTime="2026-11-03 00:32:25.134079425 +0000 UTC m=+1627.455616224"
Nov 03 00:32:25.408473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:25.408415    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:25.823009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:25.822978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:25.920877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:25.920842    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:26.101926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:26.101830    4550 generic.go:356] "Generic (PLEG): container finished" podID="a24fad59-6017-48f0-8d43-8056174747a1" containerID="bafb85bda8fdeab7b0732083c9b37c58d48ff8a349f535430714ff3115743a16" exitCode=0
Nov 03 00:32:26.101926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:26.101873    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp" event={"ID":"a24fad59-6017-48f0-8d43-8056174747a1","Type":"ContainerDied","Data":"bafb85bda8fdeab7b0732083c9b37c58d48ff8a349f535430714ff3115743a16"}
Nov 03 00:32:27.209483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.208377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:27.278073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.278050    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:27.304821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.304797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:27.356116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.356073    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr\" (UniqueName: \"kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr\") pod \"a24fad59-6017-48f0-8d43-8056174747a1\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") "
Nov 03 00:32:27.356326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.356193    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca\") pod \"a24fad59-6017-48f0-8d43-8056174747a1\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") "
Nov 03 00:32:27.356326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.356225    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca\") pod \"a24fad59-6017-48f0-8d43-8056174747a1\" (UID: \"a24fad59-6017-48f0-8d43-8056174747a1\") "
Nov 03 00:32:27.358773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.357608    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca" pod "a24fad59-6017-48f0-8d43-8056174747a1" (UID: "a24fad59-6017-48f0-8d43-8056174747a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:27.360487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.360360    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr" pod "a24fad59-6017-48f0-8d43-8056174747a1" (UID: "a24fad59-6017-48f0-8d43-8056174747a1"). InnerVolumeSpecName "kube-api-access-h5kkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:27.366345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.366312    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca" pod "a24fad59-6017-48f0-8d43-8056174747a1" (UID: "a24fad59-6017-48f0-8d43-8056174747a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:27.457861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.457827    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5kkr\" (UniqueName: \"kubernetes.io/projected/a24fad59-6017-48f0-8d43-8056174747a1-kube-api-access-h5kkr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:27.457861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.457857    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:27.457861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:27.457867    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a24fad59-6017-48f0-8d43-8056174747a1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.116169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:28.116093    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"
Nov 03 00:32:28.116169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:28.116101    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp" event={"ID":"a24fad59-6017-48f0-8d43-8056174747a1","Type":"ContainerDied","Data":"da03589b406ac4f6e331638e29174cc608e276f7fe93d9beb25df193a43b0bd8"}
Nov 03 00:32:28.116169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:28.116158    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da03589b406ac4f6e331638e29174cc608e276f7fe93d9beb25df193a43b0bd8"
Nov 03 00:32:29.034618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:29.034583    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:29.126993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:29.126718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:29.641949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:29.641917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:30.606687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:30.606649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:30.787024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:30.786988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:30.879494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:30.879402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:31.345471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:31.345422    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:33.050745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:33.050712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:33.169470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:33.169424    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:33.269729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:33.269687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:33.880300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:33.880265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:34.288176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:34.288134    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"]
Nov 03 00:32:34.288975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:34.288947    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy" containerID="cri-o://6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7" gracePeriod=30
Nov 03 00:32:34.306227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:34.306043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:35.746415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:35.746377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:35.931292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:35.931244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:36.035595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:36.035505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:36.422862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:36.422821    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:36.578651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:36.578605    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:37.616354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:37.616310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:39.480746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.480717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:39.853520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.853495    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:32:39.974948 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.974918    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.974960    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.974992    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975027    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975088    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975120    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975156    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975230    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:39.975278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975266    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975305    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket\") pod \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\" (UID: \"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e\") "
Nov 03 00:32:39.975512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975471    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:39.975635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975523    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:39.975946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975911    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:39.976066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975945    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:39.976066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975969    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:39.976066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975975    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:39.976066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.975983    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:39.977249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.977225    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:39.977552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.977524    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:39.977654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.977582    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:39.977654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:39.977628    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p" pod "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" (UID: "db1dcf41-93f4-4cfd-8a2c-f27465d0e45e"). InnerVolumeSpecName "kube-api-access-xhx6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:40.077302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.077212    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:40.077302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.077249    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:40.077302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.077264    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:40.077302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.077279    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhx6p\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-kube-api-access-xhx6p\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:40.077302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.077293    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:40.077302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.077307    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:40.166327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.166292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:40.191899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.191866    4550 generic.go:356] "Generic (PLEG): container finished" podID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerID="6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7" exitCode=0
Nov 03 00:32:40.192047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.191926    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"
Nov 03 00:32:40.192047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.191933    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" event={"ID":"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e","Type":"ContainerDied","Data":"6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7"}
Nov 03 00:32:40.192047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.192023    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94" event={"ID":"db1dcf41-93f4-4cfd-8a2c-f27465d0e45e","Type":"ContainerDied","Data":"a5af8daf97aa68f174e4a27e5cb39fcbad6714f7c9a42221d02d0e0b6d7f983d"}
Nov 03 00:32:40.192205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.192053    4550 scope.go:122] "RemoveContainer" containerID="6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7"
Nov 03 00:32:40.202802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.202778    4550 scope.go:122] "RemoveContainer" containerID="b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"
Nov 03 00:32:40.211854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.211829    4550 scope.go:122] "RemoveContainer" containerID="6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7"
Nov 03 00:32:40.212181 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:40.212162    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7\": container with ID starting with 6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7 not found: ID does not exist" containerID="6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7"
Nov 03 00:32:40.212251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.212191    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7"} err="failed to get container status \"6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7\": rpc error: code = NotFound desc = could not find container \"6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7\": container with ID starting with 6c676ce6d71c75c089fd42056b4118d008cce7e0e889a22718c135e9485c67e7 not found: ID does not exist"
Nov 03 00:32:40.212251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.212208    4550 scope.go:122] "RemoveContainer" containerID="b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"
Nov 03 00:32:40.212690 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:40.212663    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99\": container with ID starting with b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99 not found: ID does not exist" containerID="b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"
Nov 03 00:32:40.212785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.212698    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99"} err="failed to get container status \"b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99\": rpc error: code = NotFound desc = could not find container \"b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99\": container with ID starting with b7bf1781711a4219003893e4737819ffaf5c560deb0d628b2fe7a5303cdd7c99 not found: ID does not exist"
Nov 03 00:32:40.255085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.255059    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"]
Nov 03 00:32:40.265482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.265425    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:40.269820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:40.269790    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b6cd31-openshift-default-648f4789f-8kn94"]
Nov 03 00:32:41.044860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.044828    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"]
Nov 03 00:32:41.045550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.045528    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a24fad59-6017-48f0-8d43-8056174747a1" containerName="revoke-keys"
Nov 03 00:32:41.045647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.045550    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a24fad59-6017-48f0-8d43-8056174747a1" containerName="revoke-keys"
Nov 03 00:32:41.045647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.045570    4550 cpu_manager.go:414] "containerMap: removing container" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy"
Nov 03 00:32:41.045647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.045577    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy"
Nov 03 00:32:41.045814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.045705    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a24fad59-6017-48f0-8d43-8056174747a1" containerName="revoke-keys"
Nov 03 00:32:41.045814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.045719    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy"
Nov 03 00:32:41.050490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.050469    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.066017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.065992    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"]
Nov 03 00:32:41.076058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.075931    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:32:41.076058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.076014    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:32:41.191296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.191257    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nx4t\" (UniqueName: \"kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.191464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.191439    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.191576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.191559    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.292700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.292670    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.292857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.292788    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.292857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.292846    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nx4t\" (UniqueName: \"kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.293457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.293420    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.295161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.295115    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.317950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.317925    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nx4t\" (UniqueName: \"kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t\") pod \"maas-api-revoke-sub-4e67b4f6-jkfd2\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.364211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.364182    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:41.663222 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:32:41.663194    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724c8687_d7e4_4c6d_bee0_31a057617c45.slice/crio-c1298643a781d5531b39867f068406c42efcb6dc4846a845d3edf2bd375244d2 WatchSource:0}: Error finding container c1298643a781d5531b39867f068406c42efcb6dc4846a845d3edf2bd375244d2: Status 404 returned error can't find the container with id c1298643a781d5531b39867f068406c42efcb6dc4846a845d3edf2bd375244d2
Nov 03 00:32:41.663917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.663894    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"]
Nov 03 00:32:41.813016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.812986    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" path="/var/lib/kubelet/pods/db1dcf41-93f4-4cfd-8a2c-f27465d0e45e/volumes"
Nov 03 00:32:41.825384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.825355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:41.921907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:41.921836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:42.025685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:42.025659    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:42.211504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:42.211076    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2" event={"ID":"724c8687-d7e4-4c6d-bee0-31a057617c45","Type":"ContainerStarted","Data":"52cb5aaf1f0a21c96115cd148eadc1ab6ec5cc5caf7b1036fef48975d9489856"}
Nov 03 00:32:42.211504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:42.211143    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2" event={"ID":"724c8687-d7e4-4c6d-bee0-31a057617c45","Type":"ContainerStarted","Data":"c1298643a781d5531b39867f068406c42efcb6dc4846a845d3edf2bd375244d2"}
Nov 03 00:32:42.254608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:42.254551    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2" podStartSLOduration=1.254537473 podStartE2EDuration="1.254537473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:42.254215316 +0000 UTC m=+1644.575752140" watchObservedRunningTime="2026-11-03 00:32:42.254537473 +0000 UTC m=+1644.576074274"
Nov 03 00:32:43.218363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:43.218312    4550 generic.go:356] "Generic (PLEG): container finished" podID="724c8687-d7e4-4c6d-bee0-31a057617c45" containerID="52cb5aaf1f0a21c96115cd148eadc1ab6ec5cc5caf7b1036fef48975d9489856" exitCode=0
Nov 03 00:32:43.218854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:43.218400    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2" event={"ID":"724c8687-d7e4-4c6d-bee0-31a057617c45","Type":"ContainerDied","Data":"52cb5aaf1f0a21c96115cd148eadc1ab6ec5cc5caf7b1036fef48975d9489856"}
Nov 03 00:32:43.513348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:43.513272    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:44.382516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.382490    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:44.547506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.543831    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca\") pod \"724c8687-d7e4-4c6d-bee0-31a057617c45\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") "
Nov 03 00:32:44.547506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.543961    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t\" (UniqueName: \"kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t\") pod \"724c8687-d7e4-4c6d-bee0-31a057617c45\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") "
Nov 03 00:32:44.547506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.544103    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca\") pod \"724c8687-d7e4-4c6d-bee0-31a057617c45\" (UID: \"724c8687-d7e4-4c6d-bee0-31a057617c45\") "
Nov 03 00:32:44.548502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.548457    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca" pod "724c8687-d7e4-4c6d-bee0-31a057617c45" (UID: "724c8687-d7e4-4c6d-bee0-31a057617c45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:44.555025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.554864    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca" pod "724c8687-d7e4-4c6d-bee0-31a057617c45" (UID: "724c8687-d7e4-4c6d-bee0-31a057617c45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:44.555025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.554993    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t" pod "724c8687-d7e4-4c6d-bee0-31a057617c45" (UID: "724c8687-d7e4-4c6d-bee0-31a057617c45"). InnerVolumeSpecName "kube-api-access-7nx4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:44.645791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.645756    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:44.645791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.645782    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724c8687-d7e4-4c6d-bee0-31a057617c45-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:44.645791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:44.645792    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nx4t\" (UniqueName: \"kubernetes.io/projected/724c8687-d7e4-4c6d-bee0-31a057617c45-kube-api-access-7nx4t\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:45.123541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:45.123491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:45.234363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:45.234307    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2" event={"ID":"724c8687-d7e4-4c6d-bee0-31a057617c45","Type":"ContainerDied","Data":"c1298643a781d5531b39867f068406c42efcb6dc4846a845d3edf2bd375244d2"}
Nov 03 00:32:45.234363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:45.234361    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"
Nov 03 00:32:45.234623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:45.234374    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1298643a781d5531b39867f068406c42efcb6dc4846a845d3edf2bd375244d2"
Nov 03 00:32:46.837996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:46.837960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:48.322362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:48.322327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:49.180673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:49.180626    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:50.527889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:50.527859    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"]
Nov 03 00:32:50.528301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:50.528059    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy" containerID="cri-o://669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895" gracePeriod=30
Nov 03 00:32:50.913628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:50.913588    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:51.196004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:51.195919    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:51.296160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:51.296124    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:51.968044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:51.968014    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:52.341364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:52.341272    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"]
Nov 03 00:32:52.341554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:52.341522    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" containerID="cri-o://654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225" gracePeriod=30
Nov 03 00:32:52.373199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:52.373154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:52.964347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:52.964311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:53.047311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:53.047266    4550 patch_prober.go:29] interesting pod/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:32:53.047886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:53.047834    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:32:54.096897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.096863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:54.189145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.189091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:54.281007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.280947    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"]
Nov 03 00:32:54.282173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.282132    4550 cpu_manager.go:414] "containerMap: removing container" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy"
Nov 03 00:32:54.282697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.282672    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy"
Nov 03 00:32:54.282697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.282707    4550 cpu_manager.go:414] "containerMap: removing container" podUID="724c8687-d7e4-4c6d-bee0-31a057617c45" containerName="revoke-keys"
Nov 03 00:32:54.282878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.282716    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724c8687-d7e4-4c6d-bee0-31a057617c45" containerName="revoke-keys"
Nov 03 00:32:54.282944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.282901    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="db1dcf41-93f4-4cfd-8a2c-f27465d0e45e" containerName="istio-proxy"
Nov 03 00:32:54.282944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.282919    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="724c8687-d7e4-4c6d-bee0-31a057617c45" containerName="revoke-keys"
Nov 03 00:32:54.287474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.287414    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.288684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.288657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:54.306085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.306065    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"]
Nov 03 00:32:54.314598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.314574    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:32:54.314925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.314907    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:32:54.405419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.405377    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.405822 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.405793    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99c97\" (UniqueName: \"kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.405949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.405918    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.507712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.507683    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99c97\" (UniqueName: \"kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.507874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.507757    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.507937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.507872    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.508533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.508513    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.510001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.509978    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.527953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.527925    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99c97\" (UniqueName: \"kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97\") pod \"maas-api-revoke-sub-1130e749-rt45r\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.572273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.572235    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"]
Nov 03 00:32:54.577043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.577022    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.592123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.592100    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"]
Nov 03 00:32:54.608604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.608574    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl7rc\" (UniqueName: \"kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.608748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.608728    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.608862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.608839    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.620719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.620692    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:54.716884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.716791    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.717075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.716947    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.717075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.717024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl7rc\" (UniqueName: \"kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.718648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.718619    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.732692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.731290    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.749914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.749779    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl7rc\" (UniqueName: \"kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc\") pod \"maas-api-revoke-sub-63ac1c94-qk8hh\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:54.796467 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:54.794955    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:32:54.893308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:54.893275    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:55.009278 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:32:55.009248    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b8fdd9_039b_46f6_bb7d_aca26fd3624b.slice/crio-78b3254a38404a03696ee8d4154426b9ea19c4fc807f2bbd7ae34c5e92f2fec5 WatchSource:0}: Error finding container 78b3254a38404a03696ee8d4154426b9ea19c4fc807f2bbd7ae34c5e92f2fec5: Status 404 returned error can't find the container with id 78b3254a38404a03696ee8d4154426b9ea19c4fc807f2bbd7ae34c5e92f2fec5
Nov 03 00:32:55.010992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.010968    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"]
Nov 03 00:32:55.195482 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:32:55.195400    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2709b0bb_c4e7_4182_8297_d4b8fe068fa1.slice/crio-6052a35486000849218a07ed28263462a2b8b6e6e82b2a81d8780aa77cfd8006 WatchSource:0}: Error finding container 6052a35486000849218a07ed28263462a2b8b6e6e82b2a81d8780aa77cfd8006: Status 404 returned error can't find the container with id 6052a35486000849218a07ed28263462a2b8b6e6e82b2a81d8780aa77cfd8006
Nov 03 00:32:55.196685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.196662    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"]
Nov 03 00:32:55.358993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.358961    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r" event={"ID":"75b8fdd9-039b-46f6-bb7d-aca26fd3624b","Type":"ContainerStarted","Data":"7eb265fdd374293f0116b7cc51c3039cb0555192761c4aec01046545d8a3b9ec"}
Nov 03 00:32:55.359149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.359023    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r" event={"ID":"75b8fdd9-039b-46f6-bb7d-aca26fd3624b","Type":"ContainerStarted","Data":"78b3254a38404a03696ee8d4154426b9ea19c4fc807f2bbd7ae34c5e92f2fec5"}
Nov 03 00:32:55.360348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.360323    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh" event={"ID":"2709b0bb-c4e7-4182-8297-d4b8fe068fa1","Type":"ContainerStarted","Data":"6052a35486000849218a07ed28263462a2b8b6e6e82b2a81d8780aa77cfd8006"}
Nov 03 00:32:55.399647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.399601    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r" podStartSLOduration=1.399587901 podStartE2EDuration="1.399587901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:55.397558363 +0000 UTC m=+1657.719095162" watchObservedRunningTime="2026-11-03 00:32:55.399587901 +0000 UTC m=+1657.721124700"
Nov 03 00:32:55.446074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.446046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:55.718578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.713380    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:55.804887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.804860    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:32:55.932631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932597    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.932631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932635    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.932852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932664    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.932852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932692    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.932852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932810    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.932852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932838    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.933066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932890    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.933066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932929    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.933066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.932985    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.933066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933042    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs\") pod \"6922f8cf-c8bf-464d-97f3-60bec085cddd\" (UID: \"6922f8cf-c8bf-464d-97f3-60bec085cddd\") "
Nov 03 00:32:55.933279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933143    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.933279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933156    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:55.933382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933365    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.934180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933543    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.934180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933755    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.934180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933776    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6922f8cf-c8bf-464d-97f3-60bec085cddd-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.934180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933790    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.934180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933803    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.934180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.933816    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.935534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.935512    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "kube-api-access-klg67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:55.935664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.935649    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.935734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.935704    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:55.935734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:55.935702    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token" pod "6922f8cf-c8bf-464d-97f3-60bec085cddd" (UID: "6922f8cf-c8bf-464d-97f3-60bec085cddd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:56.034831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.034045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:56.034831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.034773    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.034831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.034794    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klg67\" (UniqueName: \"kubernetes.io/projected/6922f8cf-c8bf-464d-97f3-60bec085cddd-kube-api-access-klg67\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.034831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.034803    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.034831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.034812    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6922f8cf-c8bf-464d-97f3-60bec085cddd-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.367827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.367731    4550 generic.go:356] "Generic (PLEG): container finished" podID="75b8fdd9-039b-46f6-bb7d-aca26fd3624b" containerID="7eb265fdd374293f0116b7cc51c3039cb0555192761c4aec01046545d8a3b9ec" exitCode=0
Nov 03 00:32:56.368196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.367815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r" event={"ID":"75b8fdd9-039b-46f6-bb7d-aca26fd3624b","Type":"ContainerDied","Data":"7eb265fdd374293f0116b7cc51c3039cb0555192761c4aec01046545d8a3b9ec"}
Nov 03 00:32:56.369790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.369762    4550 generic.go:356] "Generic (PLEG): container finished" podID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerID="669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895" exitCode=0
Nov 03 00:32:56.369932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.369818    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"
Nov 03 00:32:56.369932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.369843    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" event={"ID":"6922f8cf-c8bf-464d-97f3-60bec085cddd","Type":"ContainerDied","Data":"669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895"}
Nov 03 00:32:56.369932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.369889    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj" event={"ID":"6922f8cf-c8bf-464d-97f3-60bec085cddd","Type":"ContainerDied","Data":"67c1783b7c99c355b0935e57edd912d600d4b3e1e0402128854a765a57eb918e"}
Nov 03 00:32:56.369932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.369916    4550 scope.go:122] "RemoveContainer" containerID="669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895"
Nov 03 00:32:56.371755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.371734    4550 generic.go:356] "Generic (PLEG): container finished" podID="2709b0bb-c4e7-4182-8297-d4b8fe068fa1" containerID="b14b204e697b9846bbd320156f29d7ee6848cd9cbcac0a700b2dae3e2411e000" exitCode=0
Nov 03 00:32:56.371865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.371850    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh" event={"ID":"2709b0bb-c4e7-4182-8297-d4b8fe068fa1","Type":"ContainerDied","Data":"b14b204e697b9846bbd320156f29d7ee6848cd9cbcac0a700b2dae3e2411e000"}
Nov 03 00:32:56.380936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.380915    4550 scope.go:122] "RemoveContainer" containerID="669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895"
Nov 03 00:32:56.381225 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:56.381207    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895\": container with ID starting with 669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895 not found: ID does not exist" containerID="669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895"
Nov 03 00:32:56.381277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.381233    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895"} err="failed to get container status \"669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895\": rpc error: code = NotFound desc = could not find container \"669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895\": container with ID starting with 669a05e454174241322fb95e6ae09b401ca347c1fb5e043e9451d36aba686895 not found: ID does not exist"
Nov 03 00:32:56.449062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.449032    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"]
Nov 03 00:32:56.460072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:56.460045    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-d8pkj"]
Nov 03 00:32:57.229898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.229864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:57.570275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.570247    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:57.573509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.573483    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:57.655286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.654970    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca\") pod \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") "
Nov 03 00:32:57.655286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.655163    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca\") pod \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") "
Nov 03 00:32:57.655286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.655272    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc\" (UniqueName: \"kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc\") pod \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") "
Nov 03 00:32:57.656305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.655302    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca\") pod \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\" (UID: \"2709b0bb-c4e7-4182-8297-d4b8fe068fa1\") "
Nov 03 00:32:57.656305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.655497    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca\") pod \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") "
Nov 03 00:32:57.656305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.655590    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97\" (UniqueName: \"kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97\") pod \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\" (UID: \"75b8fdd9-039b-46f6-bb7d-aca26fd3624b\") "
Nov 03 00:32:57.656305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.655802    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca" pod "75b8fdd9-039b-46f6-bb7d-aca26fd3624b" (UID: "75b8fdd9-039b-46f6-bb7d-aca26fd3624b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:57.656629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.656601    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca" pod "2709b0bb-c4e7-4182-8297-d4b8fe068fa1" (UID: "2709b0bb-c4e7-4182-8297-d4b8fe068fa1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:57.659235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.659210    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca" pod "75b8fdd9-039b-46f6-bb7d-aca26fd3624b" (UID: "75b8fdd9-039b-46f6-bb7d-aca26fd3624b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:57.659547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.659415    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97" pod "75b8fdd9-039b-46f6-bb7d-aca26fd3624b" (UID: "75b8fdd9-039b-46f6-bb7d-aca26fd3624b"). InnerVolumeSpecName "kube-api-access-99c97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:57.659768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.659748    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc" pod "2709b0bb-c4e7-4182-8297-d4b8fe068fa1" (UID: "2709b0bb-c4e7-4182-8297-d4b8fe068fa1"). InnerVolumeSpecName "kube-api-access-cl7rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:57.659997 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.659977    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:32:57.662143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.662105    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca" pod "2709b0bb-c4e7-4182-8297-d4b8fe068fa1" (UID: "2709b0bb-c4e7-4182-8297-d4b8fe068fa1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:57.678111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.678083    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99c97\" (UniqueName: \"kubernetes.io/projected/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-kube-api-access-99c97\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.678240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.678117    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.678240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.678132    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.678240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.678147    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl7rc\" (UniqueName: \"kubernetes.io/projected/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-kube-api-access-cl7rc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.678240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.678162    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2709b0bb-c4e7-4182-8297-d4b8fe068fa1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.678240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.678178    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b8fdd9-039b-46f6-bb7d-aca26fd3624b-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.779253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779163    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779308    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779353    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779418    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779480    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779478    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:57.779607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779511    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779539    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779571    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779610    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert\") pod \"58c940e7-9d85-4bf7-8a9d-45520f06a151\" (UID: \"58c940e7-9d85-4bf7-8a9d-45520f06a151\") "
Nov 03 00:32:57.779854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:57.779994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.779962    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:57.780205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.780186    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.780268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.780211    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.780268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.780226    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.780268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.780232    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:57.780852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.780823    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:57.781918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.781886    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "kube-api-access-p8bs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:57.782310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.782274    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:57.782394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.782343    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:57.782604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.782589    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo" pod "58c940e7-9d85-4bf7-8a9d-45520f06a151" (UID: "58c940e7-9d85-4bf7-8a9d-45520f06a151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:57.808302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.808276    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" path="/var/lib/kubelet/pods/6922f8cf-c8bf-464d-97f3-60bec085cddd/volumes"
Nov 03 00:32:57.882114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.882079    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.882331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.882297    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8bs4\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-kube-api-access-p8bs4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.882331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.882330    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.882549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.882347    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58c940e7-9d85-4bf7-8a9d-45520f06a151-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.882549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.882361    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:57.882549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:57.882374    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58c940e7-9d85-4bf7-8a9d-45520f06a151-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:32:58.387193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.387160    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"
Nov 03 00:32:58.387838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.387809    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh" event={"ID":"2709b0bb-c4e7-4182-8297-d4b8fe068fa1","Type":"ContainerDied","Data":"6052a35486000849218a07ed28263462a2b8b6e6e82b2a81d8780aa77cfd8006"}
Nov 03 00:32:58.387989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.387893    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6052a35486000849218a07ed28263462a2b8b6e6e82b2a81d8780aa77cfd8006"
Nov 03 00:32:58.390221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.390195    4550 generic.go:356] "Generic (PLEG): container finished" podID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerID="654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225" exitCode=0
Nov 03 00:32:58.390336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.390251    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"
Nov 03 00:32:58.390336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.390286    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" event={"ID":"58c940e7-9d85-4bf7-8a9d-45520f06a151","Type":"ContainerDied","Data":"654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225"}
Nov 03 00:32:58.390454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.390337    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7" event={"ID":"58c940e7-9d85-4bf7-8a9d-45520f06a151","Type":"ContainerDied","Data":"11fb21bb76769cd311e3acfac033d74a49049c27b9f82a234f24c90eafd4ed36"}
Nov 03 00:32:58.390454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.390378    4550 scope.go:122] "RemoveContainer" containerID="654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225"
Nov 03 00:32:58.391880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.391860    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r" event={"ID":"75b8fdd9-039b-46f6-bb7d-aca26fd3624b","Type":"ContainerDied","Data":"78b3254a38404a03696ee8d4154426b9ea19c4fc807f2bbd7ae34c5e92f2fec5"}
Nov 03 00:32:58.391969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.391897    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"
Nov 03 00:32:58.392034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.391903    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78b3254a38404a03696ee8d4154426b9ea19c4fc807f2bbd7ae34c5e92f2fec5"
Nov 03 00:32:58.402400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.402378    4550 scope.go:122] "RemoveContainer" containerID="16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"
Nov 03 00:32:58.411606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.411589    4550 scope.go:122] "RemoveContainer" containerID="654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225"
Nov 03 00:32:58.411863 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:58.411839    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225\": container with ID starting with 654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225 not found: ID does not exist" containerID="654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225"
Nov 03 00:32:58.411961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.411870    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225"} err="failed to get container status \"654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225\": rpc error: code = NotFound desc = could not find container \"654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225\": container with ID starting with 654eb859a9f7cc71ed3a2a649665ba10b54173ddb0ca8dc40853c45f5e1af225 not found: ID does not exist"
Nov 03 00:32:58.411961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.411887    4550 scope.go:122] "RemoveContainer" containerID="16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"
Nov 03 00:32:58.412140 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:32:58.412125    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08\": container with ID starting with 16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08 not found: ID does not exist" containerID="16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"
Nov 03 00:32:58.412191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.412140    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08"} err="failed to get container status \"16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08\": rpc error: code = NotFound desc = could not find container \"16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08\": container with ID starting with 16c6cd4b426119687b350ff931036e59db215a2daeb0286838ddb6aebc7a3b08 not found: ID does not exist"
Nov 03 00:32:58.460308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.460278    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"]
Nov 03 00:32:58.464326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.464296    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-68738d-openshift-default-f5d786d69-vv2f7"]
Nov 03 00:32:58.988633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:58.988597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:59.095371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:59.095329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:59.197112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:59.197076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:32:59.809124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:32:59.809045    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" path="/var/lib/kubelet/pods/58c940e7-9d85-4bf7-8a9d-45520f06a151/volumes"
Nov 03 00:33:00.176637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.176606    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"]
Nov 03 00:33:00.177103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177066    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2709b0bb-c4e7-4182-8297-d4b8fe068fa1" containerName="revoke-keys"
Nov 03 00:33:00.177103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177079    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2709b0bb-c4e7-4182-8297-d4b8fe068fa1" containerName="revoke-keys"
Nov 03 00:33:00.177103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177095    4550 cpu_manager.go:414] "containerMap: removing container" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy"
Nov 03 00:33:00.177103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177100    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy"
Nov 03 00:33:00.177236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177116    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy"
Nov 03 00:33:00.177236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177121    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy"
Nov 03 00:33:00.177236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177129    4550 cpu_manager.go:414] "containerMap: removing container" podUID="75b8fdd9-039b-46f6-bb7d-aca26fd3624b" containerName="revoke-keys"
Nov 03 00:33:00.177236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177136    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b8fdd9-039b-46f6-bb7d-aca26fd3624b" containerName="revoke-keys"
Nov 03 00:33:00.177236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177226    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy"
Nov 03 00:33:00.177236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177236    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6922f8cf-c8bf-464d-97f3-60bec085cddd" containerName="istio-proxy"
Nov 03 00:33:00.177422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177243    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b8fdd9-039b-46f6-bb7d-aca26fd3624b" containerName="revoke-keys"
Nov 03 00:33:00.177422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.177254    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2709b0bb-c4e7-4182-8297-d4b8fe068fa1" containerName="revoke-keys"
Nov 03 00:33:00.181933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.181915    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.198969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.198947    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"]
Nov 03 00:33:00.208292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.208271    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:00.208452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.208425    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:00.320797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.320764    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.320965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.320843    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.320965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.320883    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhrj\" (UniqueName: \"kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.422784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.422747    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.422784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.422788    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.423049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.422905    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhrj\" (UniqueName: \"kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.423444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.423403    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.425116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.425098    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.443185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.443129    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhrj\" (UniqueName: \"kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj\") pod \"maas-api-revoke-sub-307a4272-6fwv5\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.501004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.500968    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:00.801289 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:00.800474    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cb72e3_e1db_4ca2_9ff7_bd1ecfdec493.slice/crio-9a22fa6f1f59747a1a7763950c49b7d645b0779d9cb6e5776c91c1d8692b80f1 WatchSource:0}: Error finding container 9a22fa6f1f59747a1a7763950c49b7d645b0779d9cb6e5776c91c1d8692b80f1: Status 404 returned error can't find the container with id 9a22fa6f1f59747a1a7763950c49b7d645b0779d9cb6e5776c91c1d8692b80f1
Nov 03 00:33:00.801289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.801167    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"]
Nov 03 00:33:00.849543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:00.849507    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:01.168466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:01.168395    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:01.412101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:01.412053    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5" event={"ID":"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493","Type":"ContainerStarted","Data":"7fdcfd9a4ca06ce6d0bf8d458813e9bd8a41a07a14c7128f1af1ca983ed80057"}
Nov 03 00:33:01.412611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:01.412111    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5" event={"ID":"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493","Type":"ContainerStarted","Data":"9a22fa6f1f59747a1a7763950c49b7d645b0779d9cb6e5776c91c1d8692b80f1"}
Nov 03 00:33:01.452005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:01.451955    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5" podStartSLOduration=1.451942831 podStartE2EDuration="1.451942831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:01.450478311 +0000 UTC m=+1663.772015108" watchObservedRunningTime="2026-11-03 00:33:01.451942831 +0000 UTC m=+1663.773479630"
Nov 03 00:33:01.711246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:01.711213    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:02.418692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:02.418652    4550 generic.go:356] "Generic (PLEG): container finished" podID="40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" containerID="7fdcfd9a4ca06ce6d0bf8d458813e9bd8a41a07a14c7128f1af1ca983ed80057" exitCode=0
Nov 03 00:33:02.419058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:02.418697    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5" event={"ID":"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493","Type":"ContainerDied","Data":"7fdcfd9a4ca06ce6d0bf8d458813e9bd8a41a07a14c7128f1af1ca983ed80057"}
Nov 03 00:33:03.028944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.028895    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:03.591567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.591538    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:03.762991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.762893    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca\") pod \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") "
Nov 03 00:33:03.762991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.762952    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca\") pod \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") "
Nov 03 00:33:03.763218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.763113    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj\" (UniqueName: \"kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj\") pod \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\" (UID: \"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493\") "
Nov 03 00:33:03.763361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.763332    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca" pod "40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" (UID: "40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:03.763666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.763641    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:03.765771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.765747    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj" pod "40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" (UID: "40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493"). InnerVolumeSpecName "kube-api-access-gbhrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:03.765866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.765752    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca" pod "40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" (UID: "40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:03.864860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.864824    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:03.864860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:03.864855    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhrj\" (UniqueName: \"kubernetes.io/projected/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493-kube-api-access-gbhrj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:04.431886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:04.431852    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5" event={"ID":"40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493","Type":"ContainerDied","Data":"9a22fa6f1f59747a1a7763950c49b7d645b0779d9cb6e5776c91c1d8692b80f1"}
Nov 03 00:33:04.431886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:04.431877    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"
Nov 03 00:33:04.432114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:04.431904    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a22fa6f1f59747a1a7763950c49b7d645b0779d9cb6e5776c91c1d8692b80f1"
Nov 03 00:33:04.741188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:04.741104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:05.628565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.628527    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"]
Nov 03 00:33:05.629284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.629255    4550 cpu_manager.go:414] "containerMap: removing container" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy"
Nov 03 00:33:05.629284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.629284    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy"
Nov 03 00:33:05.629508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.629315    4550 cpu_manager.go:414] "containerMap: removing container" podUID="40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" containerName="revoke-keys"
Nov 03 00:33:05.629508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.629324    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" containerName="revoke-keys"
Nov 03 00:33:05.629508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.629457    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="58c940e7-9d85-4bf7-8a9d-45520f06a151" containerName="istio-proxy"
Nov 03 00:33:05.629508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.629473    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" containerName="revoke-keys"
Nov 03 00:33:05.633593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.633565    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.731414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.731373    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.732881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.732844    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733164    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733391    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733622    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733662    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733776    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkjk\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733828    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.733928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.733871    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.736263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.736241    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"]
Nov 03 00:33:05.842488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.842451    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.845615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.845585    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.845881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.845861    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.844241    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846003    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846133    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkjk\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846194    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846226    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846314    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846692    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.846887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846869    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.847029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.846997    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.850378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.850356    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.850549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.850414    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.850549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.850465    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.852672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.852646    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.924633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.924596    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkjk\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.925245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.925202    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:05.950208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:05.950184    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:06.317066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:06.315608    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"]
Nov 03 00:33:06.332060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:06.331796    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:06.332060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:06.331893    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:06.332060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:06.331941    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:06.447393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:06.447362    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" event={"ID":"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9","Type":"ContainerStarted","Data":"4ea25c1b88ed03f24b0aed5fc92d0c9db79cae1f6b4499e585d1ce74605d176f"}
Nov 03 00:33:06.781473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:06.779329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:07.455028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.454983    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" event={"ID":"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9","Type":"ContainerStarted","Data":"64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66"}
Nov 03 00:33:07.529625 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.529569    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podStartSLOduration=2.529550369 podStartE2EDuration="2.529550369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:07.514751089 +0000 UTC m=+1669.836287930" watchObservedRunningTime="2026-11-03 00:33:07.529550369 +0000 UTC m=+1669.851087170"
Nov 03 00:33:07.943124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.943088    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"]
Nov 03 00:33:07.947028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.947008    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:07.950726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.950703    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:07.952465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.952416    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:07.952580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.952478    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:07.991461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.985958    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:07.991461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.986035    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:07.991461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:07.986508    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"]
Nov 03 00:33:08.017969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.017933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:08.097592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.097544    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx85v\" (UniqueName: \"kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.097800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.097740    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.097800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.097797    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.199261 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.199166    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx85v\" (UniqueName: \"kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.199464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.199309    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.199464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.199375    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.200112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.200090    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.202712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.202690    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.219554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.219525    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx85v\" (UniqueName: \"kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v\") pod \"maas-api-revoke-sub-4439f591-wcqjt\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.296477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.296441    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:08.598969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.598871    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"]
Nov 03 00:33:08.715651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.715604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:08.950705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.950666    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:08.950887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:08.950718    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:09.502451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:09.501108    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt" event={"ID":"397dd2de-fdea-4c0c-b8e1-3a651046d0ae","Type":"ContainerStarted","Data":"1297bcb3492b3826222fe21927c1c72468e91ee9b9fcf90f6a2191743f290b0b"}
Nov 03 00:33:09.502451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:09.501168    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt" event={"ID":"397dd2de-fdea-4c0c-b8e1-3a651046d0ae","Type":"ContainerStarted","Data":"d9961ff4eb5ff55566bf53eb1319e11e0dd52c7ecf4388037f705f4a2edfa636"}
Nov 03 00:33:09.543067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:09.543015    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt" podStartSLOduration=2.543000938 podStartE2EDuration="2.543000938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:09.540303849 +0000 UTC m=+1671.861840648" watchObservedRunningTime="2026-11-03 00:33:09.543000938 +0000 UTC m=+1671.864537737"
Nov 03 00:33:09.951071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:09.951035    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:09.951489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:09.951094    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:10.509652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.509614    4550 generic.go:356] "Generic (PLEG): container finished" podID="397dd2de-fdea-4c0c-b8e1-3a651046d0ae" containerID="1297bcb3492b3826222fe21927c1c72468e91ee9b9fcf90f6a2191743f290b0b" exitCode=0
Nov 03 00:33:10.509851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.509706    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt" event={"ID":"397dd2de-fdea-4c0c-b8e1-3a651046d0ae","Type":"ContainerDied","Data":"1297bcb3492b3826222fe21927c1c72468e91ee9b9fcf90f6a2191743f290b0b"}
Nov 03 00:33:10.612965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.612928    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:10.662456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.656364    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"]
Nov 03 00:33:10.688509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.688477    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.694940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.693735    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"]
Nov 03 00:33:10.733500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.733463    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.733765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.733742    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdk2k\" (UniqueName: \"kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.733872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.733792    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.835294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.835198    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.835294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.835283    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdk2k\" (UniqueName: \"kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.835592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.835333    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.836072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.836046    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.837655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.837634    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.862461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.855931    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdk2k\" (UniqueName: \"kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k\") pod \"maas-api-revoke-sub-ade797b8-j8nvj\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:10.950956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.950921    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:10.951110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:10.950976    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:11.011296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.011260    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:11.186467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.186078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:11.353981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.353918    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"]
Nov 03 00:33:11.354864 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:11.354835    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca8f8e3_eb8c_482c_94cd_086908a3961e.slice/crio-7ebebdefb1bb0b406e13aae1d6e94a2c3745fdf6b1bf021f163505ed3df03732 WatchSource:0}: Error finding container 7ebebdefb1bb0b406e13aae1d6e94a2c3745fdf6b1bf021f163505ed3df03732: Status 404 returned error can't find the container with id 7ebebdefb1bb0b406e13aae1d6e94a2c3745fdf6b1bf021f163505ed3df03732
Nov 03 00:33:11.516416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.516382    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj" event={"ID":"2ca8f8e3-eb8c-482c-94cd-086908a3961e","Type":"ContainerStarted","Data":"7ebebdefb1bb0b406e13aae1d6e94a2c3745fdf6b1bf021f163505ed3df03732"}
Nov 03 00:33:11.750015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.749987    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:11.849304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.849272    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v\" (UniqueName: \"kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v\") pod \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") "
Nov 03 00:33:11.849425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.849404    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca\") pod \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") "
Nov 03 00:33:11.849702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.849611    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca\") pod \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\" (UID: \"397dd2de-fdea-4c0c-b8e1-3a651046d0ae\") "
Nov 03 00:33:11.849837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.849809    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca" pod "397dd2de-fdea-4c0c-b8e1-3a651046d0ae" (UID: "397dd2de-fdea-4c0c-b8e1-3a651046d0ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:11.850329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.850089    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:11.851647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.851620    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v" pod "397dd2de-fdea-4c0c-b8e1-3a651046d0ae" (UID: "397dd2de-fdea-4c0c-b8e1-3a651046d0ae"). InnerVolumeSpecName "kube-api-access-xx85v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:11.852154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.852133    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca" pod "397dd2de-fdea-4c0c-b8e1-3a651046d0ae" (UID: "397dd2de-fdea-4c0c-b8e1-3a651046d0ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:11.951411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.951369    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:11.951797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.951461    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx85v\" (UniqueName: \"kubernetes.io/projected/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-kube-api-access-xx85v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:11.951797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.951483    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397dd2de-fdea-4c0c-b8e1-3a651046d0ae-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:11.951797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:11.951485    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:12.523476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.523426    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"
Nov 03 00:33:12.523476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.523423    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt" event={"ID":"397dd2de-fdea-4c0c-b8e1-3a651046d0ae","Type":"ContainerDied","Data":"d9961ff4eb5ff55566bf53eb1319e11e0dd52c7ecf4388037f705f4a2edfa636"}
Nov 03 00:33:12.523710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.523499    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9961ff4eb5ff55566bf53eb1319e11e0dd52c7ecf4388037f705f4a2edfa636"
Nov 03 00:33:12.525080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.525049    4550 generic.go:356] "Generic (PLEG): container finished" podID="2ca8f8e3-eb8c-482c-94cd-086908a3961e" containerID="841b1e138d61d7754c2b9cbb3b4df9c375238b630343955d101d1d623d8c0f7d" exitCode=0
Nov 03 00:33:12.525221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.525197    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj" event={"ID":"2ca8f8e3-eb8c-482c-94cd-086908a3961e","Type":"ContainerDied","Data":"841b1e138d61d7754c2b9cbb3b4df9c375238b630343955d101d1d623d8c0f7d"}
Nov 03 00:33:12.950852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.950812    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:12.951041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:12.950881    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:13.097739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.097700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:13.757369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.757339    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:13.894555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.894456    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca\") pod \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") "
Nov 03 00:33:13.894733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.894590    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k\" (UniqueName: \"kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k\") pod \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") "
Nov 03 00:33:13.894733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.894623    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca\") pod \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\" (UID: \"2ca8f8e3-eb8c-482c-94cd-086908a3961e\") "
Nov 03 00:33:13.894855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.894829    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca" pod "2ca8f8e3-eb8c-482c-94cd-086908a3961e" (UID: "2ca8f8e3-eb8c-482c-94cd-086908a3961e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:13.895130 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.895105    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:13.896826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.896797    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca" pod "2ca8f8e3-eb8c-482c-94cd-086908a3961e" (UID: "2ca8f8e3-eb8c-482c-94cd-086908a3961e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:13.896905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.896831    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k" pod "2ca8f8e3-eb8c-482c-94cd-086908a3961e" (UID: "2ca8f8e3-eb8c-482c-94cd-086908a3961e"). InnerVolumeSpecName "kube-api-access-qdk2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:13.951265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.951227    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:33:13.951497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.951296    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Nov 03 00:33:13.990502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.990379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:13.996094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.996060    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdk2k\" (UniqueName: \"kubernetes.io/projected/2ca8f8e3-eb8c-482c-94cd-086908a3961e-kube-api-access-qdk2k\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:13.996094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:13.996095    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ca8f8e3-eb8c-482c-94cd-086908a3961e-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:14.090175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.090138    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:14.542612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.542568    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj" event={"ID":"2ca8f8e3-eb8c-482c-94cd-086908a3961e","Type":"ContainerDied","Data":"7ebebdefb1bb0b406e13aae1d6e94a2c3745fdf6b1bf021f163505ed3df03732"}
Nov 03 00:33:14.543331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.542635    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ebebdefb1bb0b406e13aae1d6e94a2c3745fdf6b1bf021f163505ed3df03732"
Nov 03 00:33:14.543331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.542702    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"
Nov 03 00:33:14.863664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.863589    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:14.957138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.957107    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:14.957525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.957422    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:14.958271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:14.958254    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:33:15.271954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:15.271920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:16.910815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:16.910781    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:18.306802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.306773    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"]
Nov 03 00:33:18.307444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.307407    4550 cpu_manager.go:414] "containerMap: removing container" podUID="397dd2de-fdea-4c0c-b8e1-3a651046d0ae" containerName="revoke-keys"
Nov 03 00:33:18.307551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.307445    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397dd2de-fdea-4c0c-b8e1-3a651046d0ae" containerName="revoke-keys"
Nov 03 00:33:18.307551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.307467    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2ca8f8e3-eb8c-482c-94cd-086908a3961e" containerName="revoke-keys"
Nov 03 00:33:18.307551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.307474    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca8f8e3-eb8c-482c-94cd-086908a3961e" containerName="revoke-keys"
Nov 03 00:33:18.308316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.308289    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="397dd2de-fdea-4c0c-b8e1-3a651046d0ae" containerName="revoke-keys"
Nov 03 00:33:18.308423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.308321    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca8f8e3-eb8c-482c-94cd-086908a3961e" containerName="revoke-keys"
Nov 03 00:33:18.317249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.317222    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.332226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.332203    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"]
Nov 03 00:33:18.342181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.342156    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:18.342281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.342245    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:18.455408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.455372    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmjhc\" (UniqueName: \"kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.455571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.455547    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.455610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.455587    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.556676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.556646    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.556833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.556692    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.556833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.556782    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmjhc\" (UniqueName: \"kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.557294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.557270    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.559048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.559022    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.576545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.576518    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmjhc\" (UniqueName: \"kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc\") pod \"maas-api-revoke-sub-27139631-ct6ds\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.632571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.632548    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:18.649862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.649833    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:18.939862 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:18.939835    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96570bfc_c33f_46f4_8ff8_0bfb4987559e.slice/crio-a8b4c15a7f457ff40b2e6e197bd0fa5080e2ae7e988dea5a4e081422b2317d13 WatchSource:0}: Error finding container a8b4c15a7f457ff40b2e6e197bd0fa5080e2ae7e988dea5a4e081422b2317d13: Status 404 returned error can't find the container with id a8b4c15a7f457ff40b2e6e197bd0fa5080e2ae7e988dea5a4e081422b2317d13
Nov 03 00:33:18.940036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:18.940014    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"]
Nov 03 00:33:19.580372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:19.580289    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds" event={"ID":"96570bfc-c33f-46f4-8ff8-0bfb4987559e","Type":"ContainerStarted","Data":"a840dab72d6693e119ec8b051dcfee8b6069cb8db52488073029b9b840ba8bde"}
Nov 03 00:33:19.580372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:19.580351    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds" event={"ID":"96570bfc-c33f-46f4-8ff8-0bfb4987559e","Type":"ContainerStarted","Data":"a8b4c15a7f457ff40b2e6e197bd0fa5080e2ae7e988dea5a4e081422b2317d13"}
Nov 03 00:33:19.620540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:19.620491    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds" podStartSLOduration=1.6204796460000002 podStartE2EDuration="1.620479646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:19.619812649 +0000 UTC m=+1681.941349449" watchObservedRunningTime="2026-11-03 00:33:19.620479646 +0000 UTC m=+1681.942016446"
Nov 03 00:33:20.351013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:20.350953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:20.587960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:20.587928    4550 generic.go:356] "Generic (PLEG): container finished" podID="96570bfc-c33f-46f4-8ff8-0bfb4987559e" containerID="a840dab72d6693e119ec8b051dcfee8b6069cb8db52488073029b9b840ba8bde" exitCode=0
Nov 03 00:33:20.588412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:20.588010    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds" event={"ID":"96570bfc-c33f-46f4-8ff8-0bfb4987559e","Type":"ContainerDied","Data":"a840dab72d6693e119ec8b051dcfee8b6069cb8db52488073029b9b840ba8bde"}
Nov 03 00:33:20.907067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:20.907035    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:21.752413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.752390    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:21.895978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.895872    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc\" (UniqueName: \"kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc\") pod \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") "
Nov 03 00:33:21.896351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.896327    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca\") pod \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") "
Nov 03 00:33:21.897129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.897108    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca\") pod \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\" (UID: \"96570bfc-c33f-46f4-8ff8-0bfb4987559e\") "
Nov 03 00:33:21.898840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.898813    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca" pod "96570bfc-c33f-46f4-8ff8-0bfb4987559e" (UID: "96570bfc-c33f-46f4-8ff8-0bfb4987559e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:21.905655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.905627    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc" pod "96570bfc-c33f-46f4-8ff8-0bfb4987559e" (UID: "96570bfc-c33f-46f4-8ff8-0bfb4987559e"). InnerVolumeSpecName "kube-api-access-tmjhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:21.913635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.912629    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca" pod "96570bfc-c33f-46f4-8ff8-0bfb4987559e" (UID: "96570bfc-c33f-46f4-8ff8-0bfb4987559e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:21.998946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.998902    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:21.998946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.998948    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmjhc\" (UniqueName: \"kubernetes.io/projected/96570bfc-c33f-46f4-8ff8-0bfb4987559e-kube-api-access-tmjhc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:21.999248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:21.998964    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96570bfc-c33f-46f4-8ff8-0bfb4987559e-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:22.261180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:22.261150    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:22.601123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:22.601048    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"
Nov 03 00:33:22.601123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:22.601070    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds" event={"ID":"96570bfc-c33f-46f4-8ff8-0bfb4987559e","Type":"ContainerDied","Data":"a8b4c15a7f457ff40b2e6e197bd0fa5080e2ae7e988dea5a4e081422b2317d13"}
Nov 03 00:33:22.601123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:22.601122    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8b4c15a7f457ff40b2e6e197bd0fa5080e2ae7e988dea5a4e081422b2317d13"
Nov 03 00:33:23.557566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.557530    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"]
Nov 03 00:33:23.558084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.558006    4550 cpu_manager.go:414] "containerMap: removing container" podUID="96570bfc-c33f-46f4-8ff8-0bfb4987559e" containerName="revoke-keys"
Nov 03 00:33:23.558084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.558017    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96570bfc-c33f-46f4-8ff8-0bfb4987559e" containerName="revoke-keys"
Nov 03 00:33:23.558221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.558132    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="96570bfc-c33f-46f4-8ff8-0bfb4987559e" containerName="revoke-keys"
Nov 03 00:33:23.561826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.561803    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.587171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.587142    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:23.594977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.594948    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:23.604886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.604856    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"]
Nov 03 00:33:23.720250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.720208    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.720460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.720337    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhmv\" (UniqueName: \"kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.720460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.720387    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.822099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.822011    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.822265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.822159    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.822265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.822233    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhmv\" (UniqueName: \"kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.822833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.822809    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.824423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.824402    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.842718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.842687    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhmv\" (UniqueName: \"kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv\") pod \"maas-api-revoke-sub-805491ef-zqzqc\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:23.879612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:23.879570    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:24.180865 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:24.180822    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2887ae57_4d07_4bef_bff4_525c337820a9.slice/crio-6c2986af33d936cf1c901e2ee4d89ccd3b7f13eb944f8d573f93a3da70d97a66 WatchSource:0}: Error finding container 6c2986af33d936cf1c901e2ee4d89ccd3b7f13eb944f8d573f93a3da70d97a66: Status 404 returned error can't find the container with id 6c2986af33d936cf1c901e2ee4d89ccd3b7f13eb944f8d573f93a3da70d97a66
Nov 03 00:33:24.182878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:24.182841    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"]
Nov 03 00:33:24.616951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:24.616910    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc" event={"ID":"2887ae57-4d07-4bef-bff4-525c337820a9","Type":"ContainerStarted","Data":"50d4622d6bb4a35b2086fd61bceeece723bcf029fe157a847ab16e642458e291"}
Nov 03 00:33:24.617361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:24.616978    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc" event={"ID":"2887ae57-4d07-4bef-bff4-525c337820a9","Type":"ContainerStarted","Data":"6c2986af33d936cf1c901e2ee4d89ccd3b7f13eb944f8d573f93a3da70d97a66"}
Nov 03 00:33:24.628281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:24.628247    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:24.658627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:24.658569    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc" podStartSLOduration=1.6585516230000001 podStartE2EDuration="1.658551623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:24.654943554 +0000 UTC m=+1686.976480375" watchObservedRunningTime="2026-11-03 00:33:24.658551623 +0000 UTC m=+1686.980088429"
Nov 03 00:33:24.741354 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:33:24.741264    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:33:25.624567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:25.624531    4550 generic.go:356] "Generic (PLEG): container finished" podID="2887ae57-4d07-4bef-bff4-525c337820a9" containerID="50d4622d6bb4a35b2086fd61bceeece723bcf029fe157a847ab16e642458e291" exitCode=0
Nov 03 00:33:25.624979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:25.624582    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc" event={"ID":"2887ae57-4d07-4bef-bff4-525c337820a9","Type":"ContainerDied","Data":"50d4622d6bb4a35b2086fd61bceeece723bcf029fe157a847ab16e642458e291"}
Nov 03 00:33:25.978344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:25.978311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:26.290644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.290547    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"]
Nov 03 00:33:26.317114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.317082    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:26.417038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.417001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:26.518389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.518355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:26.808106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.808081    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:26.860462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.859906    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca\") pod \"2887ae57-4d07-4bef-bff4-525c337820a9\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") "
Nov 03 00:33:26.860462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.860109    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv\" (UniqueName: \"kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv\") pod \"2887ae57-4d07-4bef-bff4-525c337820a9\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") "
Nov 03 00:33:26.860462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.860152    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca\") pod \"2887ae57-4d07-4bef-bff4-525c337820a9\" (UID: \"2887ae57-4d07-4bef-bff4-525c337820a9\") "
Nov 03 00:33:26.861849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.861600    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca" pod "2887ae57-4d07-4bef-bff4-525c337820a9" (UID: "2887ae57-4d07-4bef-bff4-525c337820a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:26.863534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.863506    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca" pod "2887ae57-4d07-4bef-bff4-525c337820a9" (UID: "2887ae57-4d07-4bef-bff4-525c337820a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:26.864217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.864182    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv" pod "2887ae57-4d07-4bef-bff4-525c337820a9" (UID: "2887ae57-4d07-4bef-bff4-525c337820a9"). InnerVolumeSpecName "kube-api-access-llhmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:26.977826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.977790    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhmv\" (UniqueName: \"kubernetes.io/projected/2887ae57-4d07-4bef-bff4-525c337820a9-kube-api-access-llhmv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:26.979787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.979740    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:26.979787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:26.979766    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2887ae57-4d07-4bef-bff4-525c337820a9-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:27.012079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.012028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:27.112536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.112502    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:27.566895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.566841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:27.640604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.640576    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"
Nov 03 00:33:27.640775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.640618    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc" event={"ID":"2887ae57-4d07-4bef-bff4-525c337820a9","Type":"ContainerDied","Data":"6c2986af33d936cf1c901e2ee4d89ccd3b7f13eb944f8d573f93a3da70d97a66"}
Nov 03 00:33:27.640775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.640705    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c2986af33d936cf1c901e2ee4d89ccd3b7f13eb944f8d573f93a3da70d97a66"
Nov 03 00:33:27.672731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:27.672704    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:28.307055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:28.307022    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:30.291676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:30.290027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:31.301457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:31.300234    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7" podUID="ebfc60c6-26a6-45d4-9450-906e7e1652e1" containerName="payload-processing" containerID="cri-o://7b1f0958323fa962130f72861c0cd610b76603fb7847272144803d06bf05db71" gracePeriod=25
Nov 03 00:33:32.216253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.216163    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:32.242517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.242481    4550 generic.go:356] "Generic (PLEG): container finished" podID="ebfc60c6-26a6-45d4-9450-906e7e1652e1" containerID="7b1f0958323fa962130f72861c0cd610b76603fb7847272144803d06bf05db71" exitCode=0
Nov 03 00:33:32.245001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.243703    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7" event={"ID":"ebfc60c6-26a6-45d4-9450-906e7e1652e1","Type":"ContainerDied","Data":"7b1f0958323fa962130f72861c0cd610b76603fb7847272144803d06bf05db71"}
Nov 03 00:33:32.291456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.275073    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:33:32.299454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.298291    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume\") pod \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") "
Nov 03 00:33:32.299454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.298365    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv\" (UniqueName: \"kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv\") pod \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\" (UID: \"ebfc60c6-26a6-45d4-9450-906e7e1652e1\") "
Nov 03 00:33:32.326500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.318234    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:32.326500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.323665    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv" pod "ebfc60c6-26a6-45d4-9450-906e7e1652e1" (UID: "ebfc60c6-26a6-45d4-9450-906e7e1652e1"). InnerVolumeSpecName "kube-api-access-mpzfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:32.326500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.323940    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume" pod "ebfc60c6-26a6-45d4-9450-906e7e1652e1" (UID: "ebfc60c6-26a6-45d4-9450-906e7e1652e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:32.406455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.401255    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebfc60c6-26a6-45d4-9450-906e7e1652e1-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:32.406455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:32.401292    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpzfv\" (UniqueName: \"kubernetes.io/projected/ebfc60c6-26a6-45d4-9450-906e7e1652e1-kube-api-access-mpzfv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:33.250496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.250467    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"
Nov 03 00:33:33.250666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.250472    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7" event={"ID":"ebfc60c6-26a6-45d4-9450-906e7e1652e1","Type":"ContainerDied","Data":"e1b969e2eb3ee1e6f9cd4916e6a9da9cb87a8714f9222ec87024cf8438962e8a"}
Nov 03 00:33:33.250666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.250590    4550 scope.go:122] "RemoveContainer" containerID="7b1f0958323fa962130f72861c0cd610b76603fb7847272144803d06bf05db71"
Nov 03 00:33:33.290364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.290335    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"]
Nov 03 00:33:33.290887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.290871    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ebfc60c6-26a6-45d4-9450-906e7e1652e1" containerName="payload-processing"
Nov 03 00:33:33.290955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.290889    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebfc60c6-26a6-45d4-9450-906e7e1652e1" containerName="payload-processing"
Nov 03 00:33:33.290955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.290903    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2887ae57-4d07-4bef-bff4-525c337820a9" containerName="revoke-keys"
Nov 03 00:33:33.290955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.290909    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2887ae57-4d07-4bef-bff4-525c337820a9" containerName="revoke-keys"
Nov 03 00:33:33.291065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.291037    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebfc60c6-26a6-45d4-9450-906e7e1652e1" containerName="payload-processing"
Nov 03 00:33:33.291065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.291049    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2887ae57-4d07-4bef-bff4-525c337820a9" containerName="revoke-keys"
Nov 03 00:33:33.295907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.295877    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.310922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.310895    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"]
Nov 03 00:33:33.313780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.313750    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"]
Nov 03 00:33:33.316576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.316551    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-191362-66ccdf96f7-tcxd7"]
Nov 03 00:33:33.323541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.323490    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:33.323541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.323502    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:33.323937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.323911    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.324050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.323976    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jf9\" (UniqueName: \"kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.324075    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.425893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.425857    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.426387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.425905    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jf9\" (UniqueName: \"kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.426387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.426034    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.426684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.426662    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.428317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.428300    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.432099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.432072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:33.445585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.445559    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jf9\" (UniqueName: \"kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9\") pod \"maas-api-revoke-sub-26c2dd8f-qj2g6\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.577368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.577281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:33.613576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.613546    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:33.809446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.809403    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebfc60c6-26a6-45d4-9450-906e7e1652e1" path="/var/lib/kubelet/pods/ebfc60c6-26a6-45d4-9450-906e7e1652e1/volumes"
Nov 03 00:33:33.897303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.897197    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:33.946617 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:33.946582    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2ba285_6b41_4c50_b155_9a7cdf04439a.slice/crio-5ead5e077ef3c5e4d3de6f8bf54c175aa8b2ef535c8c3e189451f662b3b01d03 WatchSource:0}: Error finding container 5ead5e077ef3c5e4d3de6f8bf54c175aa8b2ef535c8c3e189451f662b3b01d03: Status 404 returned error can't find the container with id 5ead5e077ef3c5e4d3de6f8bf54c175aa8b2ef535c8c3e189451f662b3b01d03
Nov 03 00:33:33.947754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:33.947733    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"]
Nov 03 00:33:34.006816 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:34.006782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:34.272669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:34.272629    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6" event={"ID":"0d2ba285-6b41-4c50-b155-9a7cdf04439a","Type":"ContainerStarted","Data":"5ead5e077ef3c5e4d3de6f8bf54c175aa8b2ef535c8c3e189451f662b3b01d03"}
Nov 03 00:33:35.281172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:35.281137    4550 generic.go:356] "Generic (PLEG): container finished" podID="0d2ba285-6b41-4c50-b155-9a7cdf04439a" containerID="4c4d2d5196f0936ff44e848b7c3f8fc9cdc1a75ecbb1d76e79d0f0800faa5e12" exitCode=0
Nov 03 00:33:35.281619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:35.281221    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6" event={"ID":"0d2ba285-6b41-4c50-b155-9a7cdf04439a","Type":"ContainerDied","Data":"4c4d2d5196f0936ff44e848b7c3f8fc9cdc1a75ecbb1d76e79d0f0800faa5e12"}
Nov 03 00:33:35.363026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:35.362992    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:36.443075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.443047    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:36.473757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.473729    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9\" (UniqueName: \"kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9\") pod \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") "
Nov 03 00:33:36.473933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.473844    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca\") pod \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") "
Nov 03 00:33:36.473985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.473930    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca\") pod \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\" (UID: \"0d2ba285-6b41-4c50-b155-9a7cdf04439a\") "
Nov 03 00:33:36.474316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.474277    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca" pod "0d2ba285-6b41-4c50-b155-9a7cdf04439a" (UID: "0d2ba285-6b41-4c50-b155-9a7cdf04439a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:36.475986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.475957    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca" pod "0d2ba285-6b41-4c50-b155-9a7cdf04439a" (UID: "0d2ba285-6b41-4c50-b155-9a7cdf04439a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:36.475986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.475960    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9" pod "0d2ba285-6b41-4c50-b155-9a7cdf04439a" (UID: "0d2ba285-6b41-4c50-b155-9a7cdf04439a"). InnerVolumeSpecName "kube-api-access-g2jf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:36.576180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.576087    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jf9\" (UniqueName: \"kubernetes.io/projected/0d2ba285-6b41-4c50-b155-9a7cdf04439a-kube-api-access-g2jf9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:36.576180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.576120    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:36.576180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.576134    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2ba285-6b41-4c50-b155-9a7cdf04439a-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:36.589042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.589000    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"]
Nov 03 00:33:36.589550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.589534    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0d2ba285-6b41-4c50-b155-9a7cdf04439a" containerName="revoke-keys"
Nov 03 00:33:36.589550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.589549    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2ba285-6b41-4c50-b155-9a7cdf04439a" containerName="revoke-keys"
Nov 03 00:33:36.589685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.589626    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2ba285-6b41-4c50-b155-9a7cdf04439a" containerName="revoke-keys"
Nov 03 00:33:36.593066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.593047    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.622512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.622478    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"]
Nov 03 00:33:36.677640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.677603    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.677815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.677730    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.677869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.677847    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.677907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.677884    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.677942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.677917    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.678048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.678030    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.678088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.678076    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.678258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.678243    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.678299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.678270    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl8pf\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.701056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.701023    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5d51a5-openshift-default-dockercfg-wxxrd\""
Nov 03 00:33:36.779084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779039    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779150    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779167    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl8pf\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779198    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779256    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779297    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779324    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779352    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779406    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779723    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779783    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.779887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.779832    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.780130 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.780103    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.780278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.780137    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.782098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.782074    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.782237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.782220    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.800106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.800076    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.800292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.800270    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl8pf\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:36.911632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.911598    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:36.914964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:36.914928    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:37.232663 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:37.232611    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698e204e_ac34_4124_9cb9_85b022651ee6.slice/crio-dfd8a4d30028eeb910c690c0a4af43cb94b1e53671cf3a5ed846948616ce6d0c WatchSource:0}: Error finding container dfd8a4d30028eeb910c690c0a4af43cb94b1e53671cf3a5ed846948616ce6d0c: Status 404 returned error can't find the container with id dfd8a4d30028eeb910c690c0a4af43cb94b1e53671cf3a5ed846948616ce6d0c
Nov 03 00:33:37.234878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.234849    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"]
Nov 03 00:33:37.235735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.235702    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:37.235818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.235766    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:37.235818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.235794    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:37.297762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.297726    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" event={"ID":"698e204e-ac34-4124-9cb9-85b022651ee6","Type":"ContainerStarted","Data":"dfd8a4d30028eeb910c690c0a4af43cb94b1e53671cf3a5ed846948616ce6d0c"}
Nov 03 00:33:37.299623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.299478    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6" event={"ID":"0d2ba285-6b41-4c50-b155-9a7cdf04439a","Type":"ContainerDied","Data":"5ead5e077ef3c5e4d3de6f8bf54c175aa8b2ef535c8c3e189451f662b3b01d03"}
Nov 03 00:33:37.299623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.299513    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"
Nov 03 00:33:37.299623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.299541    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ead5e077ef3c5e4d3de6f8bf54c175aa8b2ef535c8c3e189451f662b3b01d03"
Nov 03 00:33:37.568544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.565793    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:37.669466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:37.669403    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:38.333384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.332364    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" event={"ID":"698e204e-ac34-4124-9cb9-85b022651ee6","Type":"ContainerStarted","Data":"dc08b3e8a355ecb2d0a5a4c7bc6d05189605845feca5b29b9ecdc890a6d55269"}
Nov 03 00:33:38.396217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.396156    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podStartSLOduration=2.39613946 podStartE2EDuration="2.39613946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:38.39329725 +0000 UTC m=+1700.714834108" watchObservedRunningTime="2026-11-03 00:33:38.39613946 +0000 UTC m=+1700.717676260"
Nov 03 00:33:38.482179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.482120    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:38.887845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.887807    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"]
Nov 03 00:33:38.892701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.892673    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:38.908412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.908385    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:38.908572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.908479    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:38.910048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.910023    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"]
Nov 03 00:33:38.916593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.915628    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:38.916858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.916834    4550 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:33:38.916965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.916882    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 00:33:38.934411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.934380    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:38.934626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.934605    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:38.934751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:38.934735    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j29\" (UniqueName: \"kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.035918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.035881    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j29\" (UniqueName: \"kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.036091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.035954    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.036091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.036045    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.036811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.036791    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.038630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.038604    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.093479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.090256    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j29\" (UniqueName: \"kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29\") pod \"maas-api-revoke-sub-85622cd9-p97mp\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.217483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.217426    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:39.253063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.253010    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:39.547042 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:39.547011    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a10d957_479e_48db_a76e_563e509e40b5.slice/crio-393613cfe7108f366f31effa10781b89a05ebbc34386b1dfe9a7126fbcc3e10d WatchSource:0}: Error finding container 393613cfe7108f366f31effa10781b89a05ebbc34386b1dfe9a7126fbcc3e10d: Status 404 returned error can't find the container with id 393613cfe7108f366f31effa10781b89a05ebbc34386b1dfe9a7126fbcc3e10d
Nov 03 00:33:39.549527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.549506    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"]
Nov 03 00:33:39.584473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.579096    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:39.679459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.677490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:39.915237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.915203    4550 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:33:39.915657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:39.915257    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 00:33:40.347534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.347500    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp" event={"ID":"2a10d957-479e-48db-a76e-563e509e40b5","Type":"ContainerStarted","Data":"7cbda1ecd3f2441e5dcc38d5eddb0b296c39cabe5a421698e44cceecc8f38d0e"}
Nov 03 00:33:40.347733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.347562    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp" event={"ID":"2a10d957-479e-48db-a76e-563e509e40b5","Type":"ContainerStarted","Data":"393613cfe7108f366f31effa10781b89a05ebbc34386b1dfe9a7126fbcc3e10d"}
Nov 03 00:33:40.393866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.393813    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp" podStartSLOduration=2.393800608 podStartE2EDuration="2.393800608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:40.391590692 +0000 UTC m=+1702.713127498" watchObservedRunningTime="2026-11-03 00:33:40.393800608 +0000 UTC m=+1702.715337432"
Nov 03 00:33:40.447272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.447211    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:40.548689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.548649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:40.877456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.876959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:40.915519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.915489    4550 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:33:40.915984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:40.915545    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 00:33:41.355175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:41.355141    4550 generic.go:356] "Generic (PLEG): container finished" podID="2a10d957-479e-48db-a76e-563e509e40b5" containerID="7cbda1ecd3f2441e5dcc38d5eddb0b296c39cabe5a421698e44cceecc8f38d0e" exitCode=0
Nov 03 00:33:41.355325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:41.355210    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp" event={"ID":"2a10d957-479e-48db-a76e-563e509e40b5","Type":"ContainerDied","Data":"7cbda1ecd3f2441e5dcc38d5eddb0b296c39cabe5a421698e44cceecc8f38d0e"}
Nov 03 00:33:41.916068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:41.916033    4550 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:33:41.916468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:41.916093    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 00:33:42.587150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.587121    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:42.695525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.695492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:42.707633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.707603    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca\") pod \"2a10d957-479e-48db-a76e-563e509e40b5\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") "
Nov 03 00:33:42.707802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.707653    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca\") pod \"2a10d957-479e-48db-a76e-563e509e40b5\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") "
Nov 03 00:33:42.707802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.707720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29\" (UniqueName: \"kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29\") pod \"2a10d957-479e-48db-a76e-563e509e40b5\" (UID: \"2a10d957-479e-48db-a76e-563e509e40b5\") "
Nov 03 00:33:42.708332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.708306    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca" pod "2a10d957-479e-48db-a76e-563e509e40b5" (UID: "2a10d957-479e-48db-a76e-563e509e40b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:42.709851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.709824    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca" pod "2a10d957-479e-48db-a76e-563e509e40b5" (UID: "2a10d957-479e-48db-a76e-563e509e40b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:42.710052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.710032    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29" pod "2a10d957-479e-48db-a76e-563e509e40b5" (UID: "2a10d957-479e-48db-a76e-563e509e40b5"). InnerVolumeSpecName "kube-api-access-p4j29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:42.809647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.809580    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:42.809647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.809606    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a10d957-479e-48db-a76e-563e509e40b5-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:42.809647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.809616    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4j29\" (UniqueName: \"kubernetes.io/projected/2a10d957-479e-48db-a76e-563e509e40b5-kube-api-access-p4j29\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:42.916175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.916139    4550 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body=
Nov 03 00:33:42.916556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:42.916197    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused"
Nov 03 00:33:43.395079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:43.395039    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp" event={"ID":"2a10d957-479e-48db-a76e-563e509e40b5","Type":"ContainerDied","Data":"393613cfe7108f366f31effa10781b89a05ebbc34386b1dfe9a7126fbcc3e10d"}
Nov 03 00:33:43.395237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:43.395106    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="393613cfe7108f366f31effa10781b89a05ebbc34386b1dfe9a7126fbcc3e10d"
Nov 03 00:33:43.395237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:43.395054    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"
Nov 03 00:33:43.920462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:43.920418    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:43.920864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:43.920845    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:43.921601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:43.921579    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:33:44.349038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:44.348937    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:45.148961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:45.148914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:46.029883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:46.029846    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:47.726783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:47.726748    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:49.501343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:49.501307    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:49.605815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:49.605783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:50.668704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.668669    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"]
Nov 03 00:33:50.669216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.668917    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" podUID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" containerName="discovery" containerID="cri-o://e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13" gracePeriod=30
Nov 03 00:33:50.779365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.779324    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"]
Nov 03 00:33:50.781368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.780112    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2a10d957-479e-48db-a76e-563e509e40b5" containerName="revoke-keys"
Nov 03 00:33:50.781368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.780132    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a10d957-479e-48db-a76e-563e509e40b5" containerName="revoke-keys"
Nov 03 00:33:50.781368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.780249    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a10d957-479e-48db-a76e-563e509e40b5" containerName="revoke-keys"
Nov 03 00:33:50.808141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.802745    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.811899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.811869    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"]
Nov 03 00:33:50.941190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941106    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941160    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941401    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8vzg\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941602 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941577    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941654    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941756    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941842    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.941913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941876    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:50.942022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:50.941934    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.007926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.007743    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:33:51.042894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.042860    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.043077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.042960    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.043138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.043121    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.043218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.043188    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.043525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.043498    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.043659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.043571    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:51.043887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.043869    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.043963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.043940    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v\") pod \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\" (UID: \"92ee0c1d-6f20-4b99-a3e3-422108b84ce4\") "
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044202    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044255    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044291    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044340    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044577    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044622    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044669    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8vzg\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044703    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044712    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044783    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.044911    4550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-ca-configmap\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.045888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.045131    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.045345    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.045888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.045645    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.046069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.045926    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:51.046069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.045968    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:51.046069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.046012    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:51.046692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.046667    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:33:51.046781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.046761    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.047182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.047150    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:51.047813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.047793    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v" pod "92ee0c1d-6f20-4b99-a3e3-422108b84ce4" (UID: "92ee0c1d-6f20-4b99-a3e3-422108b84ce4"). InnerVolumeSpecName "kube-api-access-vq47v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:51.048313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.048295    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.048551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.048531    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.070330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.070292    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8vzg\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.070732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.070688    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.146677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.146647    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.146677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.146673    4550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-cacerts\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.146677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.146682    4550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-kubeconfig\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.146911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.146692    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq47v\" (UniqueName: \"kubernetes.io/projected/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-kube-api-access-vq47v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.146911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.146701    4550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-istio-csr-dns-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.146911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.146709    4550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ee0c1d-6f20-4b99-a3e3-422108b84ce4-local-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:51.151152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.151126    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:51.397034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.396991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:51.476978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.476880    4550 generic.go:356] "Generic (PLEG): container finished" podID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" containerID="e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13" exitCode=0
Nov 03 00:33:51.476978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.476932    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"
Nov 03 00:33:51.476978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.476940    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" event={"ID":"92ee0c1d-6f20-4b99-a3e3-422108b84ce4","Type":"ContainerDied","Data":"e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13"}
Nov 03 00:33:51.477274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.477016    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs" event={"ID":"92ee0c1d-6f20-4b99-a3e3-422108b84ce4","Type":"ContainerDied","Data":"054a54098cc37e7705af3efeaeb9b3097003ecf049fe4a57bcfb5cbb314775e1"}
Nov 03 00:33:51.477274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.477046    4550 scope.go:122] "RemoveContainer" containerID="e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13"
Nov 03 00:33:51.485258 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:51.485232    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5fc7b6f_1906_4411_afbc_6efaf10272b8.slice/crio-239b2efc8ca94ff642347129748dc10cd7c38bda5b314eb2145c5a3809c52a88 WatchSource:0}: Error finding container 239b2efc8ca94ff642347129748dc10cd7c38bda5b314eb2145c5a3809c52a88: Status 404 returned error can't find the container with id 239b2efc8ca94ff642347129748dc10cd7c38bda5b314eb2145c5a3809c52a88
Nov 03 00:33:51.487105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.487075    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"]
Nov 03 00:33:51.491697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.491673    4550 scope.go:122] "RemoveContainer" containerID="e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13"
Nov 03 00:33:51.492496 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:33:51.491981    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13\": container with ID starting with e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13 not found: ID does not exist" containerID="e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13"
Nov 03 00:33:51.492496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.492022    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13"} err="failed to get container status \"e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13\": rpc error: code = NotFound desc = could not find container \"e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13\": container with ID starting with e5e28238894263135be49fd4817a03509d07566e2bf2d15f56e0e88b759dfa13 not found: ID does not exist"
Nov 03 00:33:51.492668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.492523    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.492668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.492597    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.492668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.492637    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:33:51.542752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.542718    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"]
Nov 03 00:33:51.547131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.547103    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qqfrs"]
Nov 03 00:33:51.809190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.809097    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" path="/var/lib/kubelet/pods/92ee0c1d-6f20-4b99-a3e3-422108b84ce4/volumes"
Nov 03 00:33:51.821146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:51.821115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:52.497549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:52.497509    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" event={"ID":"b5fc7b6f-1906-4411-afbc-6efaf10272b8","Type":"ContainerStarted","Data":"7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38"}
Nov 03 00:33:52.497839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:52.497594    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" event={"ID":"b5fc7b6f-1906-4411-afbc-6efaf10272b8","Type":"ContainerStarted","Data":"239b2efc8ca94ff642347129748dc10cd7c38bda5b314eb2145c5a3809c52a88"}
Nov 03 00:33:52.554913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:52.554861    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podStartSLOduration=2.554847567 podStartE2EDuration="2.554847567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:52.552358007 +0000 UTC m=+1714.873894805" watchObservedRunningTime="2026-11-03 00:33:52.554847567 +0000 UTC m=+1714.876384432"
Nov 03 00:33:53.091295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.091259    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"]
Nov 03 00:33:53.091824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.091793    4550 cpu_manager.go:414] "containerMap: removing container" podUID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" containerName="discovery"
Nov 03 00:33:53.091824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.091808    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" containerName="discovery"
Nov 03 00:33:53.091959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.091912    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ee0c1d-6f20-4b99-a3e3-422108b84ce4" containerName="discovery"
Nov 03 00:33:53.100474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.095804    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.113332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.113299    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"]
Nov 03 00:33:53.115852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.115829    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:53.116420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.116403    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:53.151732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.151701    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:53.152902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.152879    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:33:53.153013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.152928    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 00:33:53.197279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.197244    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.197486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.197307    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.197537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.197493    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtnr\" (UniqueName: \"kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.250378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.250317    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:53.298851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.298806    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtnr\" (UniqueName: \"kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.299059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.299003    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.299141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.299060    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.299916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.299893    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.303112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.303088    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.323689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.323657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtnr\" (UniqueName: \"kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr\") pod \"maas-api-revoke-sub-0c0c8132-4bm9d\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.415401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.415368    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:53.747896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:53.747865    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"]
Nov 03 00:33:54.151745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:54.151715    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:33:54.152156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:54.151766    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 00:33:54.521480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:54.520345    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d" event={"ID":"6de9bcd2-da6a-46d0-952c-5daebca38c52","Type":"ContainerStarted","Data":"201d49a7414929d9347897e64d4d03e72b6fcdf031dc828efabd5496b4432305"}
Nov 03 00:33:54.521480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:54.520417    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d" event={"ID":"6de9bcd2-da6a-46d0-952c-5daebca38c52","Type":"ContainerStarted","Data":"d74fc1e1ccf8ceb95921d1aee779056fc4d119eca488f30125b7197b3d7760df"}
Nov 03 00:33:54.586451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:54.586369    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d" podStartSLOduration=1.586332979 podStartE2EDuration="1.586332979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:54.561830111 +0000 UTC m=+1716.883366936" watchObservedRunningTime="2026-11-03 00:33:54.586332979 +0000 UTC m=+1716.907869779"
Nov 03 00:33:54.852711 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:33:54.852626    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:33:55.166455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:55.160394    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:33:55.166455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:55.160482    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 00:33:55.651530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:55.626767    4550 generic.go:356] "Generic (PLEG): container finished" podID="6de9bcd2-da6a-46d0-952c-5daebca38c52" containerID="201d49a7414929d9347897e64d4d03e72b6fcdf031dc828efabd5496b4432305" exitCode=0
Nov 03 00:33:55.651530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:55.626926    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d" event={"ID":"6de9bcd2-da6a-46d0-952c-5daebca38c52","Type":"ContainerDied","Data":"201d49a7414929d9347897e64d4d03e72b6fcdf031dc828efabd5496b4432305"}
Nov 03 00:33:55.688460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:55.687922    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.066508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.018642    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.066508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.030534    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.097456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.081801    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.097456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.082152    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": context deadline exceeded"
Nov 03 00:33:56.152641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.152601    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:33:56.152921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.152893    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 00:33:56.174105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.174065    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:33:56.341508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.339518    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.373456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.372815    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.418457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.415550    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.445456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.439923    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.445633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.445524    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:56.696518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.693467    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:33:56.696518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.694230    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:33:56.798468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:56.781046    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.056520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.030836    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.123454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.082268    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:33:57.135362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.089212    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.136166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.136144    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:33:57.145634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.145613    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:33:57.150629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.150603    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:33:57.151552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.151517    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:33:57.155534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.155498    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 00:33:57.169468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.158378    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:33:57.208495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.206963    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.208495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.207592    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:33:57.436612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.434676    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.445774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.437195    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:33:57.467343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.466900    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:33:57.486909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.486292    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:33:57.644363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.644300    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:57.736563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.736532    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca\") pod \"6de9bcd2-da6a-46d0-952c-5daebca38c52\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") "
Nov 03 00:33:57.736745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.736576    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca\") pod \"6de9bcd2-da6a-46d0-952c-5daebca38c52\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") "
Nov 03 00:33:57.736745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.736695    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr\" (UniqueName: \"kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr\") pod \"6de9bcd2-da6a-46d0-952c-5daebca38c52\" (UID: \"6de9bcd2-da6a-46d0-952c-5daebca38c52\") "
Nov 03 00:33:57.740467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.740305    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca" pod "6de9bcd2-da6a-46d0-952c-5daebca38c52" (UID: "6de9bcd2-da6a-46d0-952c-5daebca38c52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:57.748772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.748729    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca" pod "6de9bcd2-da6a-46d0-952c-5daebca38c52" (UID: "6de9bcd2-da6a-46d0-952c-5daebca38c52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:57.751628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.751572    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr" pod "6de9bcd2-da6a-46d0-952c-5daebca38c52" (UID: "6de9bcd2-da6a-46d0-952c-5daebca38c52"). InnerVolumeSpecName "kube-api-access-xrtnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:57.841743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.841718    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:57.841743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.841744    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de9bcd2-da6a-46d0-952c-5daebca38c52-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:57.841899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.841754    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtnr\" (UniqueName: \"kubernetes.io/projected/6de9bcd2-da6a-46d0-952c-5daebca38c52-kube-api-access-xrtnr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:33:57.902985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.902950    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"
Nov 03 00:33:57.903140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.903013    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d" event={"ID":"6de9bcd2-da6a-46d0-952c-5daebca38c52","Type":"ContainerDied","Data":"d74fc1e1ccf8ceb95921d1aee779056fc4d119eca488f30125b7197b3d7760df"}
Nov 03 00:33:57.903140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:57.903059    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d74fc1e1ccf8ceb95921d1aee779056fc4d119eca488f30125b7197b3d7760df"
Nov 03 00:33:58.151490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:58.151456    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:33:58.151674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:58.151520    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Nov 03 00:33:58.622555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:58.622517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:33:59.233323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.233293    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"]
Nov 03 00:33:59.233821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.233807    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6de9bcd2-da6a-46d0-952c-5daebca38c52" containerName="revoke-keys"
Nov 03 00:33:59.233874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.233823    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de9bcd2-da6a-46d0-952c-5daebca38c52" containerName="revoke-keys"
Nov 03 00:33:59.233916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.233895    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de9bcd2-da6a-46d0-952c-5daebca38c52" containerName="revoke-keys"
Nov 03 00:33:59.237527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.237509    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.255395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.255372    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"]
Nov 03 00:33:59.279325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.279293    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:33:59.279725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.279708    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:33:59.359882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.359848    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nc7z\" (UniqueName: \"kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.360017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.359994    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.360126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.360111    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.460864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.460831    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.461083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.461055    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nc7z\" (UniqueName: \"kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.461170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.461146    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.461831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.461801    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.464311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.464283    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.486036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.485969    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nc7z\" (UniqueName: \"kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z\") pod \"maas-api-revoke-sub-53edfea9-rlvmm\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.552087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.552048    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:33:59.675406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.675368    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:59.676844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.676753    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:59.680276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.680256    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:33:59.885544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.885184    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"]
Nov 03 00:33:59.893937 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:33:59.893901    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf025b5_c19c_40cb_9057_a978ca5e5b64.slice/crio-7219229af3e7296d5b4d91cb0d1568feafbb48485006e7a63f2df3da801b1534 WatchSource:0}: Error finding container 7219229af3e7296d5b4d91cb0d1568feafbb48485006e7a63f2df3da801b1534: Status 404 returned error can't find the container with id 7219229af3e7296d5b4d91cb0d1568feafbb48485006e7a63f2df3da801b1534
Nov 03 00:33:59.919229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:33:59.919196    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm" event={"ID":"ecf025b5-c19c-40cb-9057-a978ca5e5b64","Type":"ContainerStarted","Data":"7219229af3e7296d5b4d91cb0d1568feafbb48485006e7a63f2df3da801b1534"}
Nov 03 00:34:00.528354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:00.528275    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:00.927863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:00.927834    4550 generic.go:356] "Generic (PLEG): container finished" podID="ecf025b5-c19c-40cb-9057-a978ca5e5b64" containerID="339d9812c3e69a07a80b5a1417d27ab0abc042fb44beb01cebdd3b165c42a956" exitCode=0
Nov 03 00:34:00.928160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:00.927918    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm" event={"ID":"ecf025b5-c19c-40cb-9057-a978ca5e5b64","Type":"ContainerDied","Data":"339d9812c3e69a07a80b5a1417d27ab0abc042fb44beb01cebdd3b165c42a956"}
Nov 03 00:34:02.104337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.104315    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:34:02.216355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.216277    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:02.230062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.230032    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca\") pod \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") "
Nov 03 00:34:02.230185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.230081    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z\" (UniqueName: \"kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z\") pod \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") "
Nov 03 00:34:02.230294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.230276    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca\") pod \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\" (UID: \"ecf025b5-c19c-40cb-9057-a978ca5e5b64\") "
Nov 03 00:34:02.230524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.230497    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca" pod "ecf025b5-c19c-40cb-9057-a978ca5e5b64" (UID: "ecf025b5-c19c-40cb-9057-a978ca5e5b64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:02.230920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.230896    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:02.232206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.232184    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z" pod "ecf025b5-c19c-40cb-9057-a978ca5e5b64" (UID: "ecf025b5-c19c-40cb-9057-a978ca5e5b64"). InnerVolumeSpecName "kube-api-access-7nc7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:02.232310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.232234    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca" pod "ecf025b5-c19c-40cb-9057-a978ca5e5b64" (UID: "ecf025b5-c19c-40cb-9057-a978ca5e5b64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:02.332966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.332929    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf025b5-c19c-40cb-9057-a978ca5e5b64-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:02.332966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.332969    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nc7z\" (UniqueName: \"kubernetes.io/projected/ecf025b5-c19c-40cb-9057-a978ca5e5b64-kube-api-access-7nc7z\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:02.943879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.943834    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm" event={"ID":"ecf025b5-c19c-40cb-9057-a978ca5e5b64","Type":"ContainerDied","Data":"7219229af3e7296d5b4d91cb0d1568feafbb48485006e7a63f2df3da801b1534"}
Nov 03 00:34:02.943879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.943885    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7219229af3e7296d5b4d91cb0d1568feafbb48485006e7a63f2df3da801b1534"
Nov 03 00:34:02.944068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:02.943848    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"
Nov 03 00:34:03.898059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:03.898029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:05.382300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:05.382249    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:05.557015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:05.556981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:05.661186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:05.661144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:06.067171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:06.067087    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:06.938371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:06.938335    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:07.044049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:07.044015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:07.937619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:07.937587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:08.112705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:08.112669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:08.315136 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:08.315040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:09.095666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:09.095632    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:10.093827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.093775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:10.329700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.329671    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"]
Nov 03 00:34:10.330176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.330162    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ecf025b5-c19c-40cb-9057-a978ca5e5b64" containerName="revoke-keys"
Nov 03 00:34:10.330231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.330177    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf025b5-c19c-40cb-9057-a978ca5e5b64" containerName="revoke-keys"
Nov 03 00:34:10.330271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.330266    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf025b5-c19c-40cb-9057-a978ca5e5b64" containerName="revoke-keys"
Nov 03 00:34:10.333369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.333353    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.351404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.351349    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"]
Nov 03 00:34:10.360726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.360700    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:10.360839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.360764    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:10.433622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.433592    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.433764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.433658    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.433810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.433761    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqxbm\" (UniqueName: \"kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.534879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.534850    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.535058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.534887    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqxbm\" (UniqueName: \"kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.535058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.535021    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.535518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.535494    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.537265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.537243    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.554681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.554659    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqxbm\" (UniqueName: \"kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm\") pod \"maas-api-revoke-sub-d4f06252-djzxc\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.645914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.645833    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:10.724921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.724884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:10.952441 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:10.952396    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bc8e04_6db0_4aa4_a641_56e79f2bc159.slice/crio-190c9f4584d3aa36b8fadc351535882e30719502028efd6e6095f430323503c2 WatchSource:0}: Error finding container 190c9f4584d3aa36b8fadc351535882e30719502028efd6e6095f430323503c2: Status 404 returned error can't find the container with id 190c9f4584d3aa36b8fadc351535882e30719502028efd6e6095f430323503c2
Nov 03 00:34:10.955005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.954981    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"]
Nov 03 00:34:10.997853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:10.997824    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc" event={"ID":"a8bc8e04-6db0-4aa4-a641-56e79f2bc159","Type":"ContainerStarted","Data":"190c9f4584d3aa36b8fadc351535882e30719502028efd6e6095f430323503c2"}
Nov 03 00:34:11.952218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:11.952178    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:12.008757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:12.008723    4550 generic.go:356] "Generic (PLEG): container finished" podID="a8bc8e04-6db0-4aa4-a641-56e79f2bc159" containerID="7cb60baaee1fac136facb5d64eb78b4c8841a763835fc02724d6dcd759f4dfec" exitCode=0
Nov 03 00:34:12.008925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:12.008784    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc" event={"ID":"a8bc8e04-6db0-4aa4-a641-56e79f2bc159","Type":"ContainerDied","Data":"7cb60baaee1fac136facb5d64eb78b4c8841a763835fc02724d6dcd759f4dfec"}
Nov 03 00:34:12.356854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:12.356818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:13.185293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.185270    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:13.268019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.267932    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca\") pod \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") "
Nov 03 00:34:13.268019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.267986    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca\") pod \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") "
Nov 03 00:34:13.268267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.268172    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm\" (UniqueName: \"kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm\") pod \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\" (UID: \"a8bc8e04-6db0-4aa4-a641-56e79f2bc159\") "
Nov 03 00:34:13.268751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.268713    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca" pod "a8bc8e04-6db0-4aa4-a641-56e79f2bc159" (UID: "a8bc8e04-6db0-4aa4-a641-56e79f2bc159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:13.271099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.271074    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca" pod "a8bc8e04-6db0-4aa4-a641-56e79f2bc159" (UID: "a8bc8e04-6db0-4aa4-a641-56e79f2bc159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:13.271218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.271198    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm" pod "a8bc8e04-6db0-4aa4-a641-56e79f2bc159" (UID: "a8bc8e04-6db0-4aa4-a641-56e79f2bc159"). InnerVolumeSpecName "kube-api-access-hqxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:13.370040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.370009    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqxbm\" (UniqueName: \"kubernetes.io/projected/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-kube-api-access-hqxbm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.370040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.370040    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.370226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.370055    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bc8e04-6db0-4aa4-a641-56e79f2bc159-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:13.441808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.441770    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"]
Nov 03 00:34:13.442467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.442445    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a8bc8e04-6db0-4aa4-a641-56e79f2bc159" containerName="revoke-keys"
Nov 03 00:34:13.442467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.442466    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bc8e04-6db0-4aa4-a641-56e79f2bc159" containerName="revoke-keys"
Nov 03 00:34:13.442640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.442628    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bc8e04-6db0-4aa4-a641-56e79f2bc159" containerName="revoke-keys"
Nov 03 00:34:13.446218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.446202    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.464598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.464570    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"]
Nov 03 00:34:13.470965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.470938    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.471174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.471156    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.471335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.471320    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khrvt\" (UniqueName: \"kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.572568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.572488    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.572568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.572531    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.572791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.572584    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khrvt\" (UniqueName: \"kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.573315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.573283    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.574801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.574771    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.592508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.592483    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khrvt\" (UniqueName: \"kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt\") pod \"maas-api-revoke-sub-96c3d74f-x4str\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:13.760582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:13.760545    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:14.022167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:14.022129    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc" event={"ID":"a8bc8e04-6db0-4aa4-a641-56e79f2bc159","Type":"ContainerDied","Data":"190c9f4584d3aa36b8fadc351535882e30719502028efd6e6095f430323503c2"}
Nov 03 00:34:14.022322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:14.022173    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"
Nov 03 00:34:14.022322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:14.022183    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="190c9f4584d3aa36b8fadc351535882e30719502028efd6e6095f430323503c2"
Nov 03 00:34:14.065315 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:14.065289    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d63564_456a_47c0_9df0_21a63ab64fc4.slice/crio-460086c7d6106aa761016c48126eb3961d2782aca465bacbf3a4756e02c27360 WatchSource:0}: Error finding container 460086c7d6106aa761016c48126eb3961d2782aca465bacbf3a4756e02c27360: Status 404 returned error can't find the container with id 460086c7d6106aa761016c48126eb3961d2782aca465bacbf3a4756e02c27360
Nov 03 00:34:14.067207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:14.067178    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"]
Nov 03 00:34:14.146273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:14.146248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:14.267452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:14.257348    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:15.029581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:15.029547    4550 generic.go:356] "Generic (PLEG): container finished" podID="c2d63564-456a-47c0-9df0-21a63ab64fc4" containerID="aa05e9c8dee4d9677811319ad135ee7caddba6ca16aedf281cdfb89e04f99e97" exitCode=0
Nov 03 00:34:15.029744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:15.029582    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str" event={"ID":"c2d63564-456a-47c0-9df0-21a63ab64fc4","Type":"ContainerDied","Data":"aa05e9c8dee4d9677811319ad135ee7caddba6ca16aedf281cdfb89e04f99e97"}
Nov 03 00:34:15.029744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:15.029632    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str" event={"ID":"c2d63564-456a-47c0-9df0-21a63ab64fc4","Type":"ContainerStarted","Data":"460086c7d6106aa761016c48126eb3961d2782aca465bacbf3a4756e02c27360"}
Nov 03 00:34:15.213501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:15.213407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:15.856411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:15.856377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:16.205330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.205303    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:16.226065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.226040    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt\" (UniqueName: \"kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt\") pod \"c2d63564-456a-47c0-9df0-21a63ab64fc4\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") "
Nov 03 00:34:16.226243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.226097    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca\") pod \"c2d63564-456a-47c0-9df0-21a63ab64fc4\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") "
Nov 03 00:34:16.226243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.226181    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca\") pod \"c2d63564-456a-47c0-9df0-21a63ab64fc4\" (UID: \"c2d63564-456a-47c0-9df0-21a63ab64fc4\") "
Nov 03 00:34:16.226588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.226559    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca" pod "c2d63564-456a-47c0-9df0-21a63ab64fc4" (UID: "c2d63564-456a-47c0-9df0-21a63ab64fc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:16.226703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.226683    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:16.228767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.228739    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca" pod "c2d63564-456a-47c0-9df0-21a63ab64fc4" (UID: "c2d63564-456a-47c0-9df0-21a63ab64fc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:16.228877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.228773    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt" pod "c2d63564-456a-47c0-9df0-21a63ab64fc4" (UID: "c2d63564-456a-47c0-9df0-21a63ab64fc4"). InnerVolumeSpecName "kube-api-access-khrvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:16.327526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.327492    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khrvt\" (UniqueName: \"kubernetes.io/projected/c2d63564-456a-47c0-9df0-21a63ab64fc4-kube-api-access-khrvt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:16.327526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:16.327526    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63564-456a-47c0-9df0-21a63ab64fc4-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:17.049650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:17.049613    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str" event={"ID":"c2d63564-456a-47c0-9df0-21a63ab64fc4","Type":"ContainerDied","Data":"460086c7d6106aa761016c48126eb3961d2782aca465bacbf3a4756e02c27360"}
Nov 03 00:34:17.050279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:17.050238    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460086c7d6106aa761016c48126eb3961d2782aca465bacbf3a4756e02c27360"
Nov 03 00:34:17.050279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:17.049637    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"
Nov 03 00:34:18.453787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:18.453750    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:19.689044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.689000    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"]
Nov 03 00:34:19.689591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.689529    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c2d63564-456a-47c0-9df0-21a63ab64fc4" containerName="revoke-keys"
Nov 03 00:34:19.689591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.689546    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d63564-456a-47c0-9df0-21a63ab64fc4" containerName="revoke-keys"
Nov 03 00:34:19.689719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.689643    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d63564-456a-47c0-9df0-21a63ab64fc4" containerName="revoke-keys"
Nov 03 00:34:19.694031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.694008    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.728400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.713925    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"]
Nov 03 00:34:19.728400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.725346    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:19.728400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.725367    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:19.728400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.725774    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rkl\" (UniqueName: \"kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.728400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.725831    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.728400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.725958    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.827272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.827236    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.827506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.827453    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.827741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.827653    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rkl\" (UniqueName: \"kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.828185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.828160    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.829739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.829720    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:19.847059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:19.847040    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rkl\" (UniqueName: \"kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl\") pod \"maas-api-revoke-sub-935eadee-lqpd8\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:20.029452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:20.029356    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:20.272109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:20.272068    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:20.347685 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:20.347650    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2715d399_6bbc_47c5_83ff_5349d2f63be0.slice/crio-3eab8ff238787de6c79d0c9bdd845acc7a976e9b165fc2c64479a884e57b55e7 WatchSource:0}: Error finding container 3eab8ff238787de6c79d0c9bdd845acc7a976e9b165fc2c64479a884e57b55e7: Status 404 returned error can't find the container with id 3eab8ff238787de6c79d0c9bdd845acc7a976e9b165fc2c64479a884e57b55e7
Nov 03 00:34:20.348491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:20.348468    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"]
Nov 03 00:34:20.485042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:20.484346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:20.584501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:20.584427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:21.094519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:21.094404    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8" event={"ID":"2715d399-6bbc-47c5-83ff-5349d2f63be0","Type":"ContainerStarted","Data":"ed324104f3d4e4b824937cc679da113a4ac1a41b65b808acc6038fb74bd50999"}
Nov 03 00:34:21.094519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:21.094481    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8" event={"ID":"2715d399-6bbc-47c5-83ff-5349d2f63be0","Type":"ContainerStarted","Data":"3eab8ff238787de6c79d0c9bdd845acc7a976e9b165fc2c64479a884e57b55e7"}
Nov 03 00:34:21.142275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:21.142217    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8" podStartSLOduration=2.142202669 podStartE2EDuration="2.142202669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:21.139891887 +0000 UTC m=+1743.461428699" watchObservedRunningTime="2026-11-03 00:34:21.142202669 +0000 UTC m=+1743.463739469"
Nov 03 00:34:22.103483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:22.103418    4550 generic.go:356] "Generic (PLEG): container finished" podID="2715d399-6bbc-47c5-83ff-5349d2f63be0" containerID="ed324104f3d4e4b824937cc679da113a4ac1a41b65b808acc6038fb74bd50999" exitCode=0
Nov 03 00:34:22.104005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:22.103505    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8" event={"ID":"2715d399-6bbc-47c5-83ff-5349d2f63be0","Type":"ContainerDied","Data":"ed324104f3d4e4b824937cc679da113a4ac1a41b65b808acc6038fb74bd50999"}
Nov 03 00:34:22.325302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:22.325269    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:23.272614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.272586    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:23.393221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.393136    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca\") pod \"2715d399-6bbc-47c5-83ff-5349d2f63be0\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") "
Nov 03 00:34:23.393402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.393317    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca\") pod \"2715d399-6bbc-47c5-83ff-5349d2f63be0\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") "
Nov 03 00:34:23.393402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.393374    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl\" (UniqueName: \"kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl\") pod \"2715d399-6bbc-47c5-83ff-5349d2f63be0\" (UID: \"2715d399-6bbc-47c5-83ff-5349d2f63be0\") "
Nov 03 00:34:23.393672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.393637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca" pod "2715d399-6bbc-47c5-83ff-5349d2f63be0" (UID: "2715d399-6bbc-47c5-83ff-5349d2f63be0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:23.393904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.393866    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:23.395535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.395506    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca" pod "2715d399-6bbc-47c5-83ff-5349d2f63be0" (UID: "2715d399-6bbc-47c5-83ff-5349d2f63be0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:23.395664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.395637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl" pod "2715d399-6bbc-47c5-83ff-5349d2f63be0" (UID: "2715d399-6bbc-47c5-83ff-5349d2f63be0"). InnerVolumeSpecName "kube-api-access-97rkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:23.495294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.495247    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rkl\" (UniqueName: \"kubernetes.io/projected/2715d399-6bbc-47c5-83ff-5349d2f63be0-kube-api-access-97rkl\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:23.495294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:23.495286    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2715d399-6bbc-47c5-83ff-5349d2f63be0-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:24.027920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:24.027883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:24.119334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:24.119254    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8" event={"ID":"2715d399-6bbc-47c5-83ff-5349d2f63be0","Type":"ContainerDied","Data":"3eab8ff238787de6c79d0c9bdd845acc7a976e9b165fc2c64479a884e57b55e7"}
Nov 03 00:34:24.119334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:24.119291    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"
Nov 03 00:34:24.119334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:24.119308    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eab8ff238787de6c79d0c9bdd845acc7a976e9b165fc2c64479a884e57b55e7"
Nov 03 00:34:24.726112 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:34:24.726075    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:34:25.827515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:25.827480    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:26.547135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:26.547095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:27.638899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:27.638865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:29.047727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.047691    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"]
Nov 03 00:34:29.048277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.048259    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2715d399-6bbc-47c5-83ff-5349d2f63be0" containerName="revoke-keys"
Nov 03 00:34:29.048345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.048281    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2715d399-6bbc-47c5-83ff-5349d2f63be0" containerName="revoke-keys"
Nov 03 00:34:29.048453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.048441    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2715d399-6bbc-47c5-83ff-5349d2f63be0" containerName="revoke-keys"
Nov 03 00:34:29.052377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.052357    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.073938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.073910    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"]
Nov 03 00:34:29.082156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.082132    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:29.082283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.082259    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:29.174204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.174169    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.174824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.174783    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhtvx\" (UniqueName: \"kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.175009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.174992    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.193823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.193797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:29.276562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.276528    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhtvx\" (UniqueName: \"kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.276743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.276573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.276743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.276612    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.277335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.277302    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.279551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.279531    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.301713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.301631    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhtvx\" (UniqueName: \"kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx\") pod \"maas-api-revoke-sub-268b17e8-fzlpg\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.349311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.349270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:29.366964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.366700    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:29.452415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.452328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:29.758909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.758884    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"]
Nov 03 00:34:29.761205 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:29.761170    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0401864d_b9b3_4dcf_a9ba_9b7fd67258cc.slice/crio-46454bc499a0b62abf42bf0f6d8a8388971975c04577b2a51bc72d4ed3291dec WatchSource:0}: Error finding container 46454bc499a0b62abf42bf0f6d8a8388971975c04577b2a51bc72d4ed3291dec: Status 404 returned error can't find the container with id 46454bc499a0b62abf42bf0f6d8a8388971975c04577b2a51bc72d4ed3291dec
Nov 03 00:34:29.968095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:29.968053    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:30.183367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:30.183327    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg" event={"ID":"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc","Type":"ContainerStarted","Data":"ab21ecceff5a480bed98b3391fd408d74ef70c825ae3bed3c99c602d208b731c"}
Nov 03 00:34:30.183896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:30.183397    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg" event={"ID":"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc","Type":"ContainerStarted","Data":"46454bc499a0b62abf42bf0f6d8a8388971975c04577b2a51bc72d4ed3291dec"}
Nov 03 00:34:30.226829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:30.226772    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg" podStartSLOduration=1.226758772 podStartE2EDuration="1.226758772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:30.222792023 +0000 UTC m=+1752.544328823" watchObservedRunningTime="2026-11-03 00:34:30.226758772 +0000 UTC m=+1752.548295572"
Nov 03 00:34:30.892096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:30.892056    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:30.997678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:30.997560    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:31.190367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:31.190333    4550 generic.go:356] "Generic (PLEG): container finished" podID="0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" containerID="ab21ecceff5a480bed98b3391fd408d74ef70c825ae3bed3c99c602d208b731c" exitCode=0
Nov 03 00:34:31.190819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:31.190424    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg" event={"ID":"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc","Type":"ContainerDied","Data":"ab21ecceff5a480bed98b3391fd408d74ef70c825ae3bed3c99c602d208b731c"}
Nov 03 00:34:31.516453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:31.513778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:31.705918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:31.705874    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:32.094181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.094087    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:32.420156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.420131    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:32.552552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.552522    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"]
Nov 03 00:34:32.553012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.552998    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" containerName="revoke-keys"
Nov 03 00:34:32.553012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.553013    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" containerName="revoke-keys"
Nov 03 00:34:32.553116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.553105    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" containerName="revoke-keys"
Nov 03 00:34:32.556564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.556544    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.572629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.572606    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca\") pod \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") "
Nov 03 00:34:32.572758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.572663    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca\") pod \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") "
Nov 03 00:34:32.572758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.572692    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx\" (UniqueName: \"kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx\") pod \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\" (UID: \"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc\") "
Nov 03 00:34:32.573279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.573256    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca" pod "0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" (UID: "0401864d-b9b3-4dcf-a9ba-9b7fd67258cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:32.574036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.574013    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:32.575945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.575923    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca" pod "0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" (UID: "0401864d-b9b3-4dcf-a9ba-9b7fd67258cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:32.578841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.577023    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"]
Nov 03 00:34:32.578841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.577069    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx" pod "0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" (UID: "0401864d-b9b3-4dcf-a9ba-9b7fd67258cc"). InnerVolumeSpecName "kube-api-access-qhtvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:32.634865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.634838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:32.674805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.674775    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk685\" (UniqueName: \"kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.674938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.674905    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.675021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.675001    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.675182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.675168    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:32.675225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.675185    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhtvx\" (UniqueName: \"kubernetes.io/projected/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc-kube-api-access-qhtvx\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:32.776631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.776595    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.776837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.776661    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.776837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.776811    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk685\" (UniqueName: \"kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.777516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.777491    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.778974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.778952    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.806024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.805996    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk685\" (UniqueName: \"kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685\") pod \"maas-api-revoke-sub-e022b265-kjx2m\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:32.872147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:32.872127    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:33.178351 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:33.178310    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708748e4_f912_4b13_8df0_60f0cf150fcd.slice/crio-52c5c3254cd8dda1f46971a0ef204cde29949bf8f44b7d9c13f02cfab2f85822 WatchSource:0}: Error finding container 52c5c3254cd8dda1f46971a0ef204cde29949bf8f44b7d9c13f02cfab2f85822: Status 404 returned error can't find the container with id 52c5c3254cd8dda1f46971a0ef204cde29949bf8f44b7d9c13f02cfab2f85822
Nov 03 00:34:33.182163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.182112    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"]
Nov 03 00:34:33.207306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.207282    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m" event={"ID":"708748e4-f912-4b13-8df0-60f0cf150fcd","Type":"ContainerStarted","Data":"52c5c3254cd8dda1f46971a0ef204cde29949bf8f44b7d9c13f02cfab2f85822"}
Nov 03 00:34:33.208547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.208530    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"
Nov 03 00:34:33.208547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.208536    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg" event={"ID":"0401864d-b9b3-4dcf-a9ba-9b7fd67258cc","Type":"ContainerDied","Data":"46454bc499a0b62abf42bf0f6d8a8388971975c04577b2a51bc72d4ed3291dec"}
Nov 03 00:34:33.208680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.208581    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46454bc499a0b62abf42bf0f6d8a8388971975c04577b2a51bc72d4ed3291dec"
Nov 03 00:34:33.210587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.210563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:33.311193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.311161    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:33.917106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:33.917071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:34.216222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:34.216145    4550 generic.go:356] "Generic (PLEG): container finished" podID="708748e4-f912-4b13-8df0-60f0cf150fcd" containerID="5eda23a45ef28e00fbadb76aabf00b52d41b4033a884db7c6792bd6c1ead8654" exitCode=0
Nov 03 00:34:34.216351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:34.216231    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m" event={"ID":"708748e4-f912-4b13-8df0-60f0cf150fcd","Type":"ContainerDied","Data":"5eda23a45ef28e00fbadb76aabf00b52d41b4033a884db7c6792bd6c1ead8654"}
Nov 03 00:34:34.317354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:34.317322    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"]
Nov 03 00:34:34.333956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:34.317792    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy" containerID="cri-o://dc08b3e8a355ecb2d0a5a4c7bc6d05189605845feca5b29b9ecdc890a6d55269" gracePeriod=30
Nov 03 00:34:34.348664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:34.348633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:34.554095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:34.554013    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:35.158066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.158036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:35.381332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.381304    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:35.454001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.453920    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca\") pod \"708748e4-f912-4b13-8df0-60f0cf150fcd\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") "
Nov 03 00:34:35.454165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.454095    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685\" (UniqueName: \"kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685\") pod \"708748e4-f912-4b13-8df0-60f0cf150fcd\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") "
Nov 03 00:34:35.454165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.454123    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca\") pod \"708748e4-f912-4b13-8df0-60f0cf150fcd\" (UID: \"708748e4-f912-4b13-8df0-60f0cf150fcd\") "
Nov 03 00:34:35.454873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.454840    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca" pod "708748e4-f912-4b13-8df0-60f0cf150fcd" (UID: "708748e4-f912-4b13-8df0-60f0cf150fcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:35.467480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.467365    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685" pod "708748e4-f912-4b13-8df0-60f0cf150fcd" (UID: "708748e4-f912-4b13-8df0-60f0cf150fcd"). InnerVolumeSpecName "kube-api-access-bk685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:35.473545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.473517    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca" pod "708748e4-f912-4b13-8df0-60f0cf150fcd" (UID: "708748e4-f912-4b13-8df0-60f0cf150fcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:35.555653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.555623    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk685\" (UniqueName: \"kubernetes.io/projected/708748e4-f912-4b13-8df0-60f0cf150fcd-kube-api-access-bk685\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:35.555653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.555654    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:35.555818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.555668    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708748e4-f912-4b13-8df0-60f0cf150fcd-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:35.769991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.769915    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"]
Nov 03 00:34:35.770699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.770678    4550 cpu_manager.go:414] "containerMap: removing container" podUID="708748e4-f912-4b13-8df0-60f0cf150fcd" containerName="revoke-keys"
Nov 03 00:34:35.770760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.770702    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708748e4-f912-4b13-8df0-60f0cf150fcd" containerName="revoke-keys"
Nov 03 00:34:35.770872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.770841    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="708748e4-f912-4b13-8df0-60f0cf150fcd" containerName="revoke-keys"
Nov 03 00:34:35.774826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.774803    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.817323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.817289    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"]
Nov 03 00:34:35.818611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.818582    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"]
Nov 03 00:34:35.824167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.822700    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.837143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.837114    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"]
Nov 03 00:34:35.859423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859394    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.859575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859495    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.859646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859582    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.859708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859653    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.859761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859741    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.859819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859806    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.859907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.859885    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87pf6\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.860061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.860015    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.860061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.860050    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.961149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961118    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.961319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961159    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.961319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961193    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.961319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961223    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgntw\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.961521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961492    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.961608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961581    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.961608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961582    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.961789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961693    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.961789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961712    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.961789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961736    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87pf6\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.961949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961802    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.961965    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.962077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962013    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962042    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962107    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962128    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.962392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962353    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.962475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962449    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:35.962532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962485    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962517    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962558    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.962888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.962866    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.964119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.964096    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.964307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.964292    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.981893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.981864    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:35.982000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:35.981928    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87pf6\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:36.063777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063710    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.063777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063755    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgntw\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.063956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063825    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.063956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063844    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.063956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063860    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.063956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063910    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.063998    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064034    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064076    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064113    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064537    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064567    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064597    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.064847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.064823    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.065979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.065959    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.066416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.066397    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.084630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.084607    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgntw\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.084747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.084730    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.089726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.089694    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:36.151205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.151184    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:36.237835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.237335    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m" event={"ID":"708748e4-f912-4b13-8df0-60f0cf150fcd","Type":"ContainerDied","Data":"52c5c3254cd8dda1f46971a0ef204cde29949bf8f44b7d9c13f02cfab2f85822"}
Nov 03 00:34:36.237835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.237405    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52c5c3254cd8dda1f46971a0ef204cde29949bf8f44b7d9c13f02cfab2f85822"
Nov 03 00:34:36.237835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.237497    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"
Nov 03 00:34:36.405469 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:36.402234    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f6ba3d_68f1_4e5f_b97d_238cc5b98411.slice/crio-4e6a5fad1cd3f480f5f2265540a3d2bf084a6df22072fdce5f45404e9913e997 WatchSource:0}: Error finding container 4e6a5fad1cd3f480f5f2265540a3d2bf084a6df22072fdce5f45404e9913e997: Status 404 returned error can't find the container with id 4e6a5fad1cd3f480f5f2265540a3d2bf084a6df22072fdce5f45404e9913e997
Nov 03 00:34:36.405469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.405020    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"]
Nov 03 00:34:36.405813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.405653    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.405813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.405730    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.405813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.405770    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.454420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.454393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:36.498887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.498849    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"]
Nov 03 00:34:36.548093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.548052    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.548235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.548125    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:36.548235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:36.548159    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:34:37.246147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:37.246113    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" event={"ID":"76f6ba3d-68f1-4e5f-b97d-238cc5b98411","Type":"ContainerStarted","Data":"33af71c9369bc9440c3424ed3bd714b6ad16348301950e311b82a2f7feef6f3d"}
Nov 03 00:34:37.246648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:37.246172    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" event={"ID":"76f6ba3d-68f1-4e5f-b97d-238cc5b98411","Type":"ContainerStarted","Data":"4e6a5fad1cd3f480f5f2265540a3d2bf084a6df22072fdce5f45404e9913e997"}
Nov 03 00:34:37.247782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:37.247758    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" event={"ID":"b5b65012-d45b-4d41-adf9-c9996ebe77db","Type":"ContainerStarted","Data":"bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57"}
Nov 03 00:34:37.247893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:37.247800    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" event={"ID":"b5b65012-d45b-4d41-adf9-c9996ebe77db","Type":"ContainerStarted","Data":"3fe1aa4bb0fc5b52bfb565969e05d4520e32727650223408ba0a2dbb78d345cc"}
Nov 03 00:34:37.342903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:37.342830    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podStartSLOduration=2.342800401 podStartE2EDuration="2.342800401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:37.296162437 +0000 UTC m=+1759.617699237" watchObservedRunningTime="2026-11-03 00:34:37.342800401 +0000 UTC m=+1759.664337202"
Nov 03 00:34:37.345153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:37.345106    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podStartSLOduration=2.345080283 podStartE2EDuration="2.345080283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:37.34161886 +0000 UTC m=+1759.663155654" watchObservedRunningTime="2026-11-03 00:34:37.345080283 +0000 UTC m=+1759.666617081"
Nov 03 00:34:38.093458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.092801    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:38.094737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.094385    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:38.094737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.094471    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:38.151488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.151449    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:38.152785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.152766    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:38.152885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.152825    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:38.245991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:38.245954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:39.109971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:39.100793    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:39.109971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:39.100860    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:39.156459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:39.154853    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:39.156459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:39.154930    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:40.100592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.100550    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:40.100905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.100869    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:40.157240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.153712    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:40.157240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.153778    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:40.157809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.157324    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.185464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.181874    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.253580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.233903    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.659289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.659146    4550 generic.go:356] "Generic (PLEG): container finished" podID="698e204e-ac34-4124-9cb9-85b022651ee6" containerID="dc08b3e8a355ecb2d0a5a4c7bc6d05189605845feca5b29b9ecdc890a6d55269" exitCode=0
Nov 03 00:34:40.659289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.659218    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" event={"ID":"698e204e-ac34-4124-9cb9-85b022651ee6","Type":"ContainerDied","Data":"dc08b3e8a355ecb2d0a5a4c7bc6d05189605845feca5b29b9ecdc890a6d55269"}
Nov 03 00:34:40.669879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.659788    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693181    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693257    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693292    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693343    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693373    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693471    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693533    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693636    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.694531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.693673    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo\") pod \"698e204e-ac34-4124-9cb9-85b022651ee6\" (UID: \"698e204e-ac34-4124-9cb9-85b022651ee6\") "
Nov 03 00:34:40.700655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.700007    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:40.700655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.700195    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:40.700655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.700360    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:40.701153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.701119    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:40.701647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.701621    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.811655    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "kube-api-access-tl8pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.812660    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl8pf\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-kube-api-access-tl8pf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.812688    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.812702    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.812716    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/698e204e-ac34-4124-9cb9-85b022651ee6-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.812730    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:40.858456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.812743    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:40.913608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.890493    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:40.995549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.890860    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:40.996702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:40.963504    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo" pod "698e204e-ac34-4124-9cb9-85b022651ee6" (UID: "698e204e-ac34-4124-9cb9-85b022651ee6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:34:41.018493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.018452    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/698e204e-ac34-4124-9cb9-85b022651ee6-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.044494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.044453    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/698e204e-ac34-4124-9cb9-85b022651ee6-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.044710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.044695    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/698e204e-ac34-4124-9cb9-85b022651ee6-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.102686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.102625    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.143470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.129493    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:41.143470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.129555    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:41.186543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.166224    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:41.186543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.166279    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:41.186543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.185498    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.186543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.186350    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:34:41.243578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.243517    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.536455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.529514    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.614657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.614596    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.615249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.615218    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.640455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.624296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:41.985606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:41.985087    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"
Nov 03 00:34:42.039578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.039520    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:42.114467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.110300    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.130909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.117261    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw" event={"ID":"698e204e-ac34-4124-9cb9-85b022651ee6","Type":"ContainerDied","Data":"dfd8a4d30028eeb910c690c0a4af43cb94b1e53671cf3a5ed846948616ce6d0c"}
Nov 03 00:34:42.130909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.124574    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:34:42.149747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.123603    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:42.150033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.150004    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:42.150160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.150146    4550 scope.go:122] "RemoveContainer" containerID="dc08b3e8a355ecb2d0a5a4c7bc6d05189605845feca5b29b9ecdc890a6d55269"
Nov 03 00:34:42.158530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.158452    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.159079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.159036    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.160488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.160469    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:42.160657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.160639    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:42.168474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.167903    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:42.279517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.252996    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.279517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.253324    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.279517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.253638    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:34:42.279517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.255171    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.437609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.437568    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.562045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.560542    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.562045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.560944    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:34:42.686309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.655194    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.686309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.655616    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:34:42.686309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.672087    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.690454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.688226    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:34:42.887967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.887928    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Nov 03 00:34:42.930689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.930638    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:42.930968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:42.930923    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.118629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.100800    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.195769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.100852    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.140516    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.185662    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.188084    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.259723    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.259785    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.217217    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.260247    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.285502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.258520    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.386583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.368801    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.386583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.369120    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:34:43.386583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.369415    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:34:43.457454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.453352    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.482468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.471989    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:34:43.482468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.481725    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:43.482468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.481775    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.507847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.507813    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"]
Nov 03 00:34:43.568029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.567994    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:34:43.610481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.603886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:43.616349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.616317    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.620551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.620516    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:43.704566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.683607    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 00:34:43.712658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:43.693703    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.110715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.110615    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:44.119947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.119812    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:44.214124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.167871    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:44.214124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.167933    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:44.243152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.243103    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8xfpw"]
Nov 03 00:34:44.243449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.243412    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:34:44.328632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.307704    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:34:44.328632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.308212    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 00:34:44.328632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.317303    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 00:34:44.401701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.401657    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.420610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.420570    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 00:34:44.458803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.458426    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:34:44.483502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.481525    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.608499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.602050    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.609129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.609097    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.734152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.734066    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.736041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.735936    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:34:44.752235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.737370    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:34:44.752235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.738618    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:34:44.752235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.739243    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:34:44.752235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.750481    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:34:44.752235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.751792    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:34:44.828049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.827717    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:34:44.829856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:44.829838    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:34:45.130981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:45.129581    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:45.130981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:45.129641    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:45.152664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:45.152633    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:45.152801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:45.152694    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:45.811404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:45.811297    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" path="/var/lib/kubelet/pods/698e204e-ac34-4124-9cb9-85b022651ee6/volumes"
Nov 03 00:34:45.945171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:45.945131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:46.090483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:46.090355    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 03 00:34:46.090483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:46.090415    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 03 00:34:46.152295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:46.152253    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 03 00:34:46.152491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:46.152309    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 03 00:34:47.009514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:47.009475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:47.107727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:47.107688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:47.579450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:47.578740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:47.865180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:47.865091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:48.093078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.092498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:48.136964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.136268    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:48.136964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.136336    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:48.152775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.152722    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:48.153087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.153030    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:48.938660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.938624    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:48.941614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.941587    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:48.944475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.944134    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:34:48.948424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.948401    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"]
Nov 03 00:34:48.949174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.949155    4550 cpu_manager.go:414] "containerMap: removing container" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy"
Nov 03 00:34:48.949241 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.949177    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy"
Nov 03 00:34:48.949305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.949293    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="698e204e-ac34-4124-9cb9-85b022651ee6" containerName="istio-proxy"
Nov 03 00:34:48.954692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.954673    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:48.980455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.971141    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"]
Nov 03 00:34:48.980455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.975720    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:48.980455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:48.975862    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:49.118597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.118557    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.119175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.118665    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.119175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.118805    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf864\" (UniqueName: \"kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.155458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.154610    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:49.155458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.154696    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:49.220498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.219633    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.220498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.219718    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf864\" (UniqueName: \"kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.220498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.219818    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.220798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.220556    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.229466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.225935    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.245123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.245094    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf864\" (UniqueName: \"kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864\") pod \"maas-api-revoke-sub-fdbe7b8e-29nr5\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.344485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.344457    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:49.385481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.383706    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:49.385642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.385518    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:49.386394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.386370    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:34:49.719498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.719470    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"]
Nov 03 00:34:49.719717 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:49.719545    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e358839_91a8_4ae2_8485_8036cd75657a.slice/crio-3663d350d5b8cc5936e8d7247d13410f37ac0cc2c725846a923a4b352658beef WatchSource:0}: Error finding container 3663d350d5b8cc5936e8d7247d13410f37ac0cc2c725846a923a4b352658beef: Status 404 returned error can't find the container with id 3663d350d5b8cc5936e8d7247d13410f37ac0cc2c725846a923a4b352658beef
Nov 03 00:34:49.769279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:49.769192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:50.408365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:50.408331    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5" event={"ID":"9e358839-91a8-4ae2-8485-8036cd75657a","Type":"ContainerStarted","Data":"d9a72c10b180f38def910aba1d456eeaeb4956160be135d609830e88e6b12b97"}
Nov 03 00:34:50.408859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:50.408391    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5" event={"ID":"9e358839-91a8-4ae2-8485-8036cd75657a","Type":"ContainerStarted","Data":"3663d350d5b8cc5936e8d7247d13410f37ac0cc2c725846a923a4b352658beef"}
Nov 03 00:34:50.450122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:50.450066    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5" podStartSLOduration=2.450050757 podStartE2EDuration="2.450050757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:50.446798207 +0000 UTC m=+1772.768335032" watchObservedRunningTime="2026-11-03 00:34:50.450050757 +0000 UTC m=+1772.771587557"
Nov 03 00:34:51.416172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:51.416139    4550 generic.go:356] "Generic (PLEG): container finished" podID="9e358839-91a8-4ae2-8485-8036cd75657a" containerID="d9a72c10b180f38def910aba1d456eeaeb4956160be135d609830e88e6b12b97" exitCode=0
Nov 03 00:34:51.416632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:51.416234    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5" event={"ID":"9e358839-91a8-4ae2-8485-8036cd75657a","Type":"ContainerDied","Data":"d9a72c10b180f38def910aba1d456eeaeb4956160be135d609830e88e6b12b97"}
Nov 03 00:34:51.652121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:51.652083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:51.771033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:51.770942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:51.888304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:51.888274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:52.588539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.588510    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:52.692690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.692595    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca\") pod \"9e358839-91a8-4ae2-8485-8036cd75657a\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") "
Nov 03 00:34:52.692690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.692692    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864\" (UniqueName: \"kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864\") pod \"9e358839-91a8-4ae2-8485-8036cd75657a\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") "
Nov 03 00:34:52.693015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.692722    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca\") pod \"9e358839-91a8-4ae2-8485-8036cd75657a\" (UID: \"9e358839-91a8-4ae2-8485-8036cd75657a\") "
Nov 03 00:34:52.693155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.693127    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca" pod "9e358839-91a8-4ae2-8485-8036cd75657a" (UID: "9e358839-91a8-4ae2-8485-8036cd75657a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:52.694934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.694911    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864" pod "9e358839-91a8-4ae2-8485-8036cd75657a" (UID: "9e358839-91a8-4ae2-8485-8036cd75657a"). InnerVolumeSpecName "kube-api-access-tf864". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:52.694934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.694920    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca" pod "9e358839-91a8-4ae2-8485-8036cd75657a" (UID: "9e358839-91a8-4ae2-8485-8036cd75657a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:52.794414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.794378    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:52.794414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.794410    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e358839-91a8-4ae2-8485-8036cd75657a-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:52.794414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:52.794419    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf864\" (UniqueName: \"kubernetes.io/projected/9e358839-91a8-4ae2-8485-8036cd75657a-kube-api-access-tf864\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:53.430810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:53.430775    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"
Nov 03 00:34:53.430810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:53.430785    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5" event={"ID":"9e358839-91a8-4ae2-8485-8036cd75657a","Type":"ContainerDied","Data":"3663d350d5b8cc5936e8d7247d13410f37ac0cc2c725846a923a4b352658beef"}
Nov 03 00:34:53.431074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:53.430855    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3663d350d5b8cc5936e8d7247d13410f37ac0cc2c725846a923a4b352658beef"
Nov 03 00:34:53.515806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:53.515774    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:53.618680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:53.618641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:53.756847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:53.756690    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:54.671263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.671229    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"]
Nov 03 00:34:54.671772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.671758    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9e358839-91a8-4ae2-8485-8036cd75657a" containerName="revoke-keys"
Nov 03 00:34:54.671824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.671774    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e358839-91a8-4ae2-8485-8036cd75657a" containerName="revoke-keys"
Nov 03 00:34:54.671861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.671849    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e358839-91a8-4ae2-8485-8036cd75657a" containerName="revoke-keys"
Nov 03 00:34:54.675495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.675441    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.694615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.694578    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"]
Nov 03 00:34:54.701841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.701787    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:34:54.701841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.701837    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:34:54.739483 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:34:54.739426    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:34:54.746616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.746583    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.746818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.746793    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b9m\" (UniqueName: \"kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.747006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.746963    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.848800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.848759    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.849023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.848843    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.849023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.848997    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b9m\" (UniqueName: \"kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.849875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.849849    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.851323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.851297    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.869250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.869225    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b9m\" (UniqueName: \"kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m\") pod \"maas-api-revoke-sub-43f82238-kb74s\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:54.991731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:54.991641    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:55.294687 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:34:55.294658    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4852a3f_2afd_4a75_812a_d3bce55a5e31.slice/crio-3f4a769637ec29e35a8343bc98e66ec73e3f045a71ad4bf2d49aead0cbbd8670 WatchSource:0}: Error finding container 3f4a769637ec29e35a8343bc98e66ec73e3f045a71ad4bf2d49aead0cbbd8670: Status 404 returned error can't find the container with id 3f4a769637ec29e35a8343bc98e66ec73e3f045a71ad4bf2d49aead0cbbd8670
Nov 03 00:34:55.298738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:55.298711    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"]
Nov 03 00:34:55.331124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:55.331086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:55.449579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:55.449519    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s" event={"ID":"f4852a3f-2afd-4a75-812a-d3bce55a5e31","Type":"ContainerStarted","Data":"3f4a769637ec29e35a8343bc98e66ec73e3f045a71ad4bf2d49aead0cbbd8670"}
Nov 03 00:34:56.457245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:56.457195    4550 generic.go:356] "Generic (PLEG): container finished" podID="f4852a3f-2afd-4a75-812a-d3bce55a5e31" containerID="f7b3d40bbef545b16fcaaac3ef199f90fc1f47118ead92e9346b8fbc0717c14b" exitCode=0
Nov 03 00:34:56.457776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:56.457309    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s" event={"ID":"f4852a3f-2afd-4a75-812a-d3bce55a5e31","Type":"ContainerDied","Data":"f7b3d40bbef545b16fcaaac3ef199f90fc1f47118ead92e9346b8fbc0717c14b"}
Nov 03 00:34:56.834682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:56.834595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:57.633180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.633155    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:57.685284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.685254    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca\") pod \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") "
Nov 03 00:34:57.685487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.685292    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m\" (UniqueName: \"kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m\") pod \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") "
Nov 03 00:34:57.686045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.686010    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca" pod "f4852a3f-2afd-4a75-812a-d3bce55a5e31" (UID: "f4852a3f-2afd-4a75-812a-d3bce55a5e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:57.688268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.688226    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m" pod "f4852a3f-2afd-4a75-812a-d3bce55a5e31" (UID: "f4852a3f-2afd-4a75-812a-d3bce55a5e31"). InnerVolumeSpecName "kube-api-access-w7b9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:57.688708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.688682    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca\") pod \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\" (UID: \"f4852a3f-2afd-4a75-812a-d3bce55a5e31\") "
Nov 03 00:34:57.689234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.689210    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:57.689326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.689236    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7b9m\" (UniqueName: \"kubernetes.io/projected/f4852a3f-2afd-4a75-812a-d3bce55a5e31-kube-api-access-w7b9m\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:57.691407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.691371    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca" pod "f4852a3f-2afd-4a75-812a-d3bce55a5e31" (UID: "f4852a3f-2afd-4a75-812a-d3bce55a5e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:57.723058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.722977    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:34:57.791282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:57.791242    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4852a3f-2afd-4a75-812a-d3bce55a5e31-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:34:58.474154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:58.474128    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"
Nov 03 00:34:58.474372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:58.474163    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s" event={"ID":"f4852a3f-2afd-4a75-812a-d3bce55a5e31","Type":"ContainerDied","Data":"3f4a769637ec29e35a8343bc98e66ec73e3f045a71ad4bf2d49aead0cbbd8670"}
Nov 03 00:34:58.474372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:58.474225    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f4a769637ec29e35a8343bc98e66ec73e3f045a71ad4bf2d49aead0cbbd8670"
Nov 03 00:34:59.448594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:34:59.448557    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:00.193635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:00.193592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:00.297890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:00.297838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:00.670829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:00.670786    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:00.800417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:00.800378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:00.922196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:00.922111    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:01.520470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:01.520443    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:02.395455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:02.392175    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:02.494063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:02.493936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:03.682456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:03.675474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:04.382472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.381556    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.509451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.497343    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:04.601870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.598684    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.750461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.748743    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.810454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.801593    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Nov 03 00:35:04.995638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.995596    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:35:04.995931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:04.995905    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.360398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.360342    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.368678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.360744    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.368678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.362499    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.368678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.362559    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.368678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.364037    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.508007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.507966    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.508522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.508496    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.508945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.508926    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:35:05.614795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.614706    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.626770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.626742    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:35:05.770869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.770818    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.771651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:05.771630    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:35:06.068942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.068541    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.314802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.314762    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"]
Nov 03 00:35:06.315293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.315259    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy" containerID="cri-o://5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208" gracePeriod=30
Nov 03 00:35:06.366855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.366816    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.380552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.367006    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.381879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.367029    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.384970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.367055    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.475299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.381246    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:35:06.617506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.544120    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.617506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.544206    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.617506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.544251    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.635464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.618701    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.637576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.637544    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:35:06.689457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.664939    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:35:06.739609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.739580    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:35:06.739878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.739861    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:35:06.740007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.739993    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"]
Nov 03 00:35:06.740304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.740277    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" containerID="cri-o://01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8" gracePeriod=30
Nov 03 00:35:06.744452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.728543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:06.855473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.794044    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.855473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.795287    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.855473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.825729    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.956456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:06.930090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:07.332423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.332324    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded"
Nov 03 00:35:07.386225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.386155    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.386620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.386587    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 00:35:07.387873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.387826    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.555564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.539294    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.599786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.598579    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:07.655957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.655909    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.669813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.669772    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.671487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.671462    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:35:07.698405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.698362    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.700030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.699995    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.700713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.700686    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:35:07.758658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.722263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:07.802456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.795546    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.802456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.798630    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:35:07.829375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:07.825078    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": context deadline exceeded"
Nov 03 00:35:08.063606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.063562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:08.125936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.125904    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 00:35:08.237094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.237055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:08.375800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.375708    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:08.391583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.391531    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.393228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.391814    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 00:35:08.409025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.408986    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.451577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.412757    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 00:35:08.526084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.526053    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 00:35:08.562627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.562598    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 00:35:08.570653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.570626    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 00:35:08.696456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.658083    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.696456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.679336    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.773873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.740853    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.825648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.825606    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.849853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.842040    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 00:35:08.864758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.861724    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 00:35:08.912134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.912107    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 00:35:08.955885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.955808    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 00:35:08.966623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:08.966596    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 00:35:09.631150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:09.631113    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:09.751039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:09.750986    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:10.021114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:10.021071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:10.120497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:10.120467    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:10.137001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:10.136969    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:35:10.137155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:10.137027    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:35:10.853676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:10.853637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:11.214593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.214550    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:11.620827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.620802    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:35:11.699523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699488    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699601    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699663    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699754    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699752    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.699831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699804    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699864    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.699988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699878    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699907    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.699988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.699940    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.700182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700005    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo\") pod \"4c133b69-7e30-4bac-b992-cbfeed98e428\" (UID: \"4c133b69-7e30-4bac-b992-cbfeed98e428\") "
Nov 03 00:35:11.700182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700080    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.700572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700549    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.700711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700478    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:11.700816 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700557    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.700962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700826    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.700962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.700845    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.702601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.702575    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.704691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.704654    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "kube-api-access-mblk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:11.704973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.704956    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:11.705561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.705538    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo" pod "4c133b69-7e30-4bac-b992-cbfeed98e428" (UID: "4c133b69-7e30-4bac-b992-cbfeed98e428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:35:11.801925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.801835    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblk4\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-kube-api-access-mblk4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.802090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.801935    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.802090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.801956    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c133b69-7e30-4bac-b992-cbfeed98e428-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.802090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.801967    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.802090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.801978    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.802090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.801989    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c133b69-7e30-4bac-b992-cbfeed98e428-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.968490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:11.968455    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:12.068095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.068010    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:12.360601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.360571    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:35:12.470172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.470139    4550 generic.go:356] "Generic (PLEG): container finished" podID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerID="01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8" exitCode=0
Nov 03 00:35:12.470347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.470209    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"
Nov 03 00:35:12.470347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.470216    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" event={"ID":"661be69b-5f30-4354-ab07-3cfefb26c64d","Type":"ContainerDied","Data":"01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8"}
Nov 03 00:35:12.470347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.470267    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf" event={"ID":"661be69b-5f30-4354-ab07-3cfefb26c64d","Type":"ContainerDied","Data":"39eb9b54d88c2ce7d2ee02e8cb3c8630fa807e5193e1d60db6e6ac7b37a9390f"}
Nov 03 00:35:12.470347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.470298    4550 scope.go:122] "RemoveContainer" containerID="01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8"
Nov 03 00:35:12.471765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.471739    4550 generic.go:356] "Generic (PLEG): container finished" podID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerID="5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208" exitCode=0
Nov 03 00:35:12.471842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.471790    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"
Nov 03 00:35:12.471842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.471818    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" event={"ID":"4c133b69-7e30-4bac-b992-cbfeed98e428","Type":"ContainerDied","Data":"5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208"}
Nov 03 00:35:12.471918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.471847    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp" event={"ID":"4c133b69-7e30-4bac-b992-cbfeed98e428","Type":"ContainerDied","Data":"35c61277f43bda8e069f8bb6fbdf793eb1176dbb6cf3420a56895a2da8e5d689"}
Nov 03 00:35:12.480312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.480292    4550 scope.go:122] "RemoveContainer" containerID="01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8"
Nov 03 00:35:12.480589 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:35:12.480569    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8\": container with ID starting with 01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8 not found: ID does not exist" containerID="01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8"
Nov 03 00:35:12.480645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.480599    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8"} err="failed to get container status \"01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8\": rpc error: code = NotFound desc = could not find container \"01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8\": container with ID starting with 01d3be92de1a3ac83e5607ac952edd9f4319857f2dd340307d2026c3bdcdace8 not found: ID does not exist"
Nov 03 00:35:12.480645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.480618    4550 scope.go:122] "RemoveContainer" containerID="5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208"
Nov 03 00:35:12.489325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.489305    4550 scope.go:122] "RemoveContainer" containerID="5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208"
Nov 03 00:35:12.489635 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:35:12.489616    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208\": container with ID starting with 5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208 not found: ID does not exist" containerID="5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208"
Nov 03 00:35:12.489711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.489641    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208"} err="failed to get container status \"5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208\": rpc error: code = NotFound desc = could not find container \"5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208\": container with ID starting with 5b255f1371f13858f000870c5bba1d4d7c5902c2c9e4acfc3038edf5b8544208 not found: ID does not exist"
Nov 03 00:35:12.523866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.523829    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.524087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.523888    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.524087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.523958    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.524087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524016    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.524995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524120    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.524995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524199    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.524995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524225    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:12.527510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524252    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.527633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524339    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:12.527633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524471    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:12.527633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524631    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:12.527633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.524793    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:12.527633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.527118    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:12.527633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.527589    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.527927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.527711    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token\") pod \"661be69b-5f30-4354-ab07-3cfefb26c64d\" (UID: \"661be69b-5f30-4354-ab07-3cfefb26c64d\") "
Nov 03 00:35:12.527927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.527810    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"]
Nov 03 00:35:12.528191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.528174    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.528293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.528278    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.528359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.528297    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/661be69b-5f30-4354-ab07-3cfefb26c64d-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.528359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.528310    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.528359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.528322    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.528359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.528334    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.530574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.530548    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:35:12.530684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.530567    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "kube-api-access-svfn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:12.531194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.531173    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token" pod "661be69b-5f30-4354-ab07-3cfefb26c64d" (UID: "661be69b-5f30-4354-ab07-3cfefb26c64d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:12.532047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.532025    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-sc5jp"]
Nov 03 00:35:12.629738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.629656    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svfn8\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-kube-api-access-svfn8\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.629738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.629687    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.629738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.629700    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/661be69b-5f30-4354-ab07-3cfefb26c64d-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:12.822148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.822113    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"]
Nov 03 00:35:12.827645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:12.827611    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-smttf"]
Nov 03 00:35:13.767795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:13.767760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:13.809248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:13.809209    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" path="/var/lib/kubelet/pods/4c133b69-7e30-4bac-b992-cbfeed98e428/volumes"
Nov 03 00:35:13.809890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:13.809873    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" path="/var/lib/kubelet/pods/661be69b-5f30-4354-ab07-3cfefb26c64d/volumes"
Nov 03 00:35:15.485067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:15.485025    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:17.238506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.238468    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:17.660713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.660683    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"]
Nov 03 00:35:17.661281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661261    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy"
Nov 03 00:35:17.661281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661281    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy"
Nov 03 00:35:17.661484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661306    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f4852a3f-2afd-4a75-812a-d3bce55a5e31" containerName="revoke-keys"
Nov 03 00:35:17.661484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661313    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4852a3f-2afd-4a75-812a-d3bce55a5e31" containerName="revoke-keys"
Nov 03 00:35:17.661484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661334    4550 cpu_manager.go:414] "containerMap: removing container" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy"
Nov 03 00:35:17.661484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661341    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy"
Nov 03 00:35:17.661712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661492    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c133b69-7e30-4bac-b992-cbfeed98e428" containerName="istio-proxy"
Nov 03 00:35:17.661712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661508    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4852a3f-2afd-4a75-812a-d3bce55a5e31" containerName="revoke-keys"
Nov 03 00:35:17.661712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.661522    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="661be69b-5f30-4354-ab07-3cfefb26c64d" containerName="istio-proxy"
Nov 03 00:35:17.666571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.666549    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.682830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.682800    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"]
Nov 03 00:35:17.694733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.694709    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:35:17.694902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.694733    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:35:17.797145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.797108    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.797338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.797230    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.797492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.797473    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qkr\" (UniqueName: \"kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.903232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.903177    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qkr\" (UniqueName: \"kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.903417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.903254    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.903417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.903326    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.904262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.904237    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.918911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.918813    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:35:17.930105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.930032    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.949741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.948152    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qkr\" (UniqueName: \"kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr\") pod \"maas-api-revoke-sub-a8511496-vxzfm\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:17.997967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:17.997939    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:35:18.004060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:18.004035    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:18.306565 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:35:18.306533    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad94918f_9834_4b99_91a0_d06ce448bd14.slice/crio-6a650a89b2cff61dab088460806b4f8004088ea2d62761605c9939ac3cdd44dd WatchSource:0}: Error finding container 6a650a89b2cff61dab088460806b4f8004088ea2d62761605c9939ac3cdd44dd: Status 404 returned error can't find the container with id 6a650a89b2cff61dab088460806b4f8004088ea2d62761605c9939ac3cdd44dd
Nov 03 00:35:18.307976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:18.307953    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"]
Nov 03 00:35:18.518480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:18.518445    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm" event={"ID":"ad94918f-9834-4b99-91a0-d06ce448bd14","Type":"ContainerStarted","Data":"6a650a89b2cff61dab088460806b4f8004088ea2d62761605c9939ac3cdd44dd"}
Nov 03 00:35:19.454201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:19.453941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:19.539661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:19.533117    4550 generic.go:356] "Generic (PLEG): container finished" podID="ad94918f-9834-4b99-91a0-d06ce448bd14" containerID="7475aef6335feb0296dd20a4c9e1ec4e55e8cd2cb28223de6628128e6cf4a89d" exitCode=0
Nov 03 00:35:19.539661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:19.533355    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm" event={"ID":"ad94918f-9834-4b99-91a0-d06ce448bd14","Type":"ContainerDied","Data":"7475aef6335feb0296dd20a4c9e1ec4e55e8cd2cb28223de6628128e6cf4a89d"}
Nov 03 00:35:20.247454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.246603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:20.658336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.657335    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"]
Nov 03 00:35:20.702996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.683338    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.702996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.702241    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"]
Nov 03 00:35:20.733221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.726123    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c59124-openshift-default-dockercfg-7f78f\""
Nov 03 00:35:20.736263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.735928    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:20.785154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.784846    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.785154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.784939    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.785154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.785016    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.785154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.785059    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.785154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.785107    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.789638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.789209    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvx4d\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.789638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.789272    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.789638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.789362    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.789638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.789393    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.915475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.912849    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.915475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.912940    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.915475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.913010    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.915475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.913062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.929493    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvx4d\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.929579    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.929670    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.929700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.929740    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.930206    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.931481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.931360    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.947158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.944048    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.947158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.946657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.947158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.947113    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.959530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.959473    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:20.974511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.974411    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:21.020386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:20.992575    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:21.025363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.024173    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvx4d\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:21.037177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.037047    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"]
Nov 03 00:35:21.043046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.043024    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.064201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.057654    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:21.064201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.061184    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"]
Nov 03 00:35:21.088656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.072770    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"]
Nov 03 00:35:21.088656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.072868    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.088656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.077479    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:21.089112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.088752    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:21.094222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.094200    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"]
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233364    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca\") pod \"ad94918f-9834-4b99-91a0-d06ce448bd14\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") "
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233578    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr\" (UniqueName: \"kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr\") pod \"ad94918f-9834-4b99-91a0-d06ce448bd14\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") "
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233631    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca\") pod \"ad94918f-9834-4b99-91a0-d06ce448bd14\" (UID: \"ad94918f-9834-4b99-91a0-d06ce448bd14\") "
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233816    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233900    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233931    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.233967    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234019    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234056    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdnm\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234107    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234131    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234167    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234190    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234216    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234265    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234307    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234334    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234384    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234415    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234476    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rlk\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.234890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.234502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.236858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.236827    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca" pod "ad94918f-9834-4b99-91a0-d06ce448bd14" (UID: "ad94918f-9834-4b99-91a0-d06ce448bd14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:21.238413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.238251    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr" pod "ad94918f-9834-4b99-91a0-d06ce448bd14" (UID: "ad94918f-9834-4b99-91a0-d06ce448bd14"). InnerVolumeSpecName "kube-api-access-47qkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:21.242510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.241672    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca" pod "ad94918f-9834-4b99-91a0-d06ce448bd14" (UID: "ad94918f-9834-4b99-91a0-d06ce448bd14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:35:21.336333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336294    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.336549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336354    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.336549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336409    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.336549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336461    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdnm\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.336549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336512    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.336549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336538    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.336788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336570    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.336788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336592    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.336788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336620    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.336943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336787    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336948    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337009    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.336951    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337044    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337096    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337141    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337192    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rlk\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337228    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337242    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337308    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337331    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337409    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337515    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qkr\" (UniqueName: \"kubernetes.io/projected/ad94918f-9834-4b99-91a0-d06ce448bd14-kube-api-access-47qkr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:21.337596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337517    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.337596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337533    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:21.337596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337550    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad94918f-9834-4b99-91a0-d06ce448bd14-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:35:21.337854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337819    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337333    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.337985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.337962    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.338107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.338080    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.338177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.338138    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.339642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.339616    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.339751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.339616    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.339838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.339822    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.339909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.339887    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.357181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.357126    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.357344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.357313    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.357496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.357479    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdnm\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.357588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.357506    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rlk\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.416603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.416532    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:21.427072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.427045    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:21.436843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.436821    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"]
Nov 03 00:35:21.443658 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:35:21.443620    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3204b61_751e_49c9_bcdb_007c8f5206f4.slice/crio-fd9aa460f3d79fbc7986f5d55358739dd7423ff10eaffd8519337a5e9c669891 WatchSource:0}: Error finding container fd9aa460f3d79fbc7986f5d55358739dd7423ff10eaffd8519337a5e9c669891: Status 404 returned error can't find the container with id fd9aa460f3d79fbc7986f5d55358739dd7423ff10eaffd8519337a5e9c669891
Nov 03 00:35:21.446474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.446401    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.446562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.446524    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.447230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.446891    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.576097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.576044    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"
Nov 03 00:35:21.581033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.577095    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm" event={"ID":"ad94918f-9834-4b99-91a0-d06ce448bd14","Type":"ContainerDied","Data":"6a650a89b2cff61dab088460806b4f8004088ea2d62761605c9939ac3cdd44dd"}
Nov 03 00:35:21.581323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.581303    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a650a89b2cff61dab088460806b4f8004088ea2d62761605c9939ac3cdd44dd"
Nov 03 00:35:21.590264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.590230    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" event={"ID":"c3204b61-751e-49c9-bcdb-007c8f5206f4","Type":"ContainerStarted","Data":"b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"}
Nov 03 00:35:21.592232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.592207    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" event={"ID":"c3204b61-751e-49c9-bcdb-007c8f5206f4","Type":"ContainerStarted","Data":"fd9aa460f3d79fbc7986f5d55358739dd7423ff10eaffd8519337a5e9c669891"}
Nov 03 00:35:21.648497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.647762    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podStartSLOduration=1.647741646 podStartE2EDuration="1.647741646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:21.639424406 +0000 UTC m=+1803.960961210" watchObservedRunningTime="2026-11-03 00:35:21.647741646 +0000 UTC m=+1803.969278445"
Nov 03 00:35:21.773554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.773514    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"]
Nov 03 00:35:21.774133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.774113    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"]
Nov 03 00:35:21.775939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.774520    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.775939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.774620    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.775939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.774668    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.800217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.800168    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.800507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.800481    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:21.800685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:21.800665    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:22.060259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.060172    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:22.067202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.067174    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:22.145033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.144997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:22.237932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.237894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:22.600805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.600751    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" event={"ID":"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195","Type":"ContainerStarted","Data":"6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b"}
Nov 03 00:35:22.601023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.600814    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" event={"ID":"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195","Type":"ContainerStarted","Data":"c10fb4cbabfd31ddcad0bdb635af8512b7a5e29bb38cc0cbe0369bdc34a30d55"}
Nov 03 00:35:22.602911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.602876    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" event={"ID":"5baea46a-4f16-487c-9606-b3eb9e9ed71e","Type":"ContainerStarted","Data":"93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059"}
Nov 03 00:35:22.603043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.602930    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" event={"ID":"5baea46a-4f16-487c-9606-b3eb9e9ed71e","Type":"ContainerStarted","Data":"a3f9f944e0a418cfdf29fdd6912b96fa34fa8147f9d08a5768646dbac25d902a"}
Nov 03 00:35:22.603315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.603273    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:22.604236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.604219    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:35:22.647786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:22.647727    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podStartSLOduration=1.647708103 podStartE2EDuration="1.647708103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:22.643525202 +0000 UTC m=+1804.965062023" watchObservedRunningTime="2026-11-03 00:35:22.647708103 +0000 UTC m=+1804.969244902"
Nov 03 00:35:23.424147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:23.424113    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:23.424839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:23.424813    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:23.425008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:23.424980    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:23.437128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:23.437096    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:23.437479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:23.437456    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:23.437634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:23.437613    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:24.137286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.137242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:24.227015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.226945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:24.331461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.331412    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:24.420456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.418811    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:24.420456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.418871    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:24.434455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.430773    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:24.434455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.430831    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:24.481518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.469102    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:24.590465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:24.589506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:24.751595 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:35:24.751506    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:35:25.418642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:25.417938    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:25.418642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:25.418024    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:25.438454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:25.427850    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:25.438454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:25.427935    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:26.278455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.265746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:26.416029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.415990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:26.420776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.420016    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:26.420776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.420069    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:26.428706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.428509    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:26.428706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.428598    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:26.429567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.429527    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podStartSLOduration=6.429512778 podStartE2EDuration="6.429512778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:22.739339547 +0000 UTC m=+1805.060876347" watchObservedRunningTime="2026-11-03 00:35:26.429512778 +0000 UTC m=+1808.751049577"
Nov 03 00:35:26.431495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.431476    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"]
Nov 03 00:35:26.432036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.432021    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ad94918f-9834-4b99-91a0-d06ce448bd14" containerName="revoke-keys"
Nov 03 00:35:26.432111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.432037    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad94918f-9834-4b99-91a0-d06ce448bd14" containerName="revoke-keys"
Nov 03 00:35:26.432170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.432116    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad94918f-9834-4b99-91a0-d06ce448bd14" containerName="revoke-keys"
Nov 03 00:35:26.435876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.435851    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:26.475457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.455873    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"]
Nov 03 00:35:26.475457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.463637    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c59124\""
Nov 03 00:35:26.475457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.464232    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c59124-dockercfg-hmq82\""
Nov 03 00:35:26.542232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.542143    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:26.652459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.640727    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:26.652459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.640788    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmfqd\" (UniqueName: \"kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd\") pod \"payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:26.826498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.779534    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:26.826783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.826751    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmfqd\" (UniqueName: \"kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd\") pod \"payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:26.844506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.820583    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:26.924122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:26.924092    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmfqd\" (UniqueName: \"kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd\") pod \"payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:27.053057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.053023    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:27.427457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.426345    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:27.427457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.426391    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:27.427766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.427737    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:27.427825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.427801    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:27.524981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.524835    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"]
Nov 03 00:35:27.853846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.853756    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh" event={"ID":"189586c0-e6ba-4806-8742-1054cbadf17a","Type":"ContainerStarted","Data":"04c2d1caad0004c610a4a0805db5dd65528df46484c0081927da26874bf8237d"}
Nov 03 00:35:27.853846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.853811    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh" event={"ID":"189586c0-e6ba-4806-8742-1054cbadf17a","Type":"ContainerStarted","Data":"6a6c3270b17dca009974374316bca3f99a1685b57a3b5be7eae4a6bc431aec20"}
Nov 03 00:35:27.854095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.853906    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:27.923481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:27.915907    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh" podStartSLOduration=1.91588892 podStartE2EDuration="1.91588892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:27.91274993 +0000 UTC m=+1810.234286731" watchObservedRunningTime="2026-11-03 00:35:27.91588892 +0000 UTC m=+1810.237425721"
Nov 03 00:35:28.395455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:28.394943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:28.426242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:28.425813    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:28.426242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:28.425874    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:28.438292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:28.438255    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:28.438583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:28.438552    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:28.927848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:28.927812    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:29.349094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.349011    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"]
Nov 03 00:35:29.354934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.354909    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.399256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.395743    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9908e8-openshift-default-dockercfg-mk7t5\""
Nov 03 00:35:29.399256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.397071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:29.422454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.422367    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"]
Nov 03 00:35:29.448454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.445774    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 00:35:29.448454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.445827    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 00:35:29.448454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.446413    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body=
Nov 03 00:35:29.448454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.446466    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458538    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r2cz\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458665    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458700    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458742    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458833    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458863    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458893    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.458926    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.460009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.459005    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562233    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562337    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r2cz\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562448    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562481    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562518    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562602    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562633    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562664    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.562698    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.574744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.563149    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.606096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.605333    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.606096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.605636    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.606690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.606621    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.607700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.607584    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.618771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.618710    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.691159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.690820    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r2cz\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.704482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.703447    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.721896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.715105    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy\") pod \"e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") " pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.744787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.744208    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:29.864508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:29.860929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:30.116941 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:35:30.116870    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6f197d_8e10_4bee_a8a0_748069364c01.slice/crio-41829a67d333667ec2a6ba5f87b75fbb723ae3f0b6dc08d7f8815e04d820e8e1 WatchSource:0}: Error finding container 41829a67d333667ec2a6ba5f87b75fbb723ae3f0b6dc08d7f8815e04d820e8e1: Status 404 returned error can't find the container with id 41829a67d333667ec2a6ba5f87b75fbb723ae3f0b6dc08d7f8815e04d820e8e1
Nov 03 00:35:30.120883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.120859    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"]
Nov 03 00:35:30.121082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.121043    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:30.121170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.121135    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:30.121240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.121181    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:30.364313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.364276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:30.566457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.565744    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:30.573451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.567426    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:30.577458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.574063    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:35:30.774552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.774514    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:30.777711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.777622    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:30.780152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.780083    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:35:30.893483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.893365    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" event={"ID":"6c6f197d-8e10-4bee-a8a0-748069364c01","Type":"ContainerStarted","Data":"b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"}
Nov 03 00:35:30.893483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.893412    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" event={"ID":"6c6f197d-8e10-4bee-a8a0-748069364c01","Type":"ContainerStarted","Data":"41829a67d333667ec2a6ba5f87b75fbb723ae3f0b6dc08d7f8815e04d820e8e1"}
Nov 03 00:35:30.941934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:30.938993    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podStartSLOduration=1.938974064 podStartE2EDuration="1.938974064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:30.938871423 +0000 UTC m=+1813.260408226" watchObservedRunningTime="2026-11-03 00:35:30.938974064 +0000 UTC m=+1813.260510866"
Nov 03 00:35:31.745676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:31.745642    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:31.751215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:31.751188    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:31.899808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:31.899782    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:31.900611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:31.900588    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:35:32.205077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:32.205044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:32.428315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:32.428279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:32.676044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:32.676003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:33.293480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:33.290963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:33.451058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:33.451029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:33.665191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:33.665160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:33.794457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:33.792982    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:34.926190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:34.926157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:35.328921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:35.328848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:35.421832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:35.421797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:36.089614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.089578    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"]
Nov 03 00:35:36.094078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.094055    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.124924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.124897    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.124968    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125115    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125201    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125245    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125272    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk8sj\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125395    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125442    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.125577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.125561    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.127522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.127501    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"]
Nov 03 00:35:36.226736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.226701    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.226751    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk8sj\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.226795    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.226814    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.226859    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.226930    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227013    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227076    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227127    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227136    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227204    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227726    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227766    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.227873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.227778    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.229604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.229579    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.231159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.231139    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.247596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.247568    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.247720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.247574    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk8sj\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.412623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.412591    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:36.728133 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:35:36.728098    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5fbc04_e9ea_47c0_ae97_b82a8bda8522.slice/crio-bbc691093d3eb655097018840e0dc49dafb4b06b9b835b83cfa4084904f3e99c WatchSource:0}: Error finding container bbc691093d3eb655097018840e0dc49dafb4b06b9b835b83cfa4084904f3e99c: Status 404 returned error can't find the container with id bbc691093d3eb655097018840e0dc49dafb4b06b9b835b83cfa4084904f3e99c
Nov 03 00:35:36.729924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.729903    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"]
Nov 03 00:35:36.730757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.730727    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:36.730853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.730790    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:36.730853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.730822    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:36.938124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.938088    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" event={"ID":"de5fbc04-e9ea-47c0-ae97-b82a8bda8522","Type":"ContainerStarted","Data":"77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08"}
Nov 03 00:35:36.938307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.938149    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" event={"ID":"de5fbc04-e9ea-47c0-ae97-b82a8bda8522","Type":"ContainerStarted","Data":"bbc691093d3eb655097018840e0dc49dafb4b06b9b835b83cfa4084904f3e99c"}
Nov 03 00:35:36.982790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:36.982682    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podStartSLOduration=0.982670099 podStartE2EDuration="982.670099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:36.981102825 +0000 UTC m=+1819.302639626" watchObservedRunningTime="2026-11-03 00:35:36.982670099 +0000 UTC m=+1819.304206970"
Nov 03 00:35:37.092947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:37.092896    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:37.413165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:37.413137    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:37.414483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:37.414452    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:37.414591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:37.414510    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:37.879508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:37.878657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:37.982457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:37.981570    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:38.413190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:38.413156    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:38.413625 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:38.413214    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:38.887981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:38.887901    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:35:39.097026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.096991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:39.415862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.415815    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:39.416345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.415881    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:39.590464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.590409    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:39.703499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.703103    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:39.834733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.820043    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"]
Nov 03 00:35:39.834733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.829867    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"]
Nov 03 00:35:39.834733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.829989    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:39.847550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.847518    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-7ef1b6\""
Nov 03 00:35:39.897462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.893753    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-7ef1b6-dockercfg-g9gsv\""
Nov 03 00:35:39.932362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.932319    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:39.932764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:39.932740    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvc9r\" (UniqueName: \"kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r\") pod \"payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:40.038400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.038290    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:40.038741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.038719    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvc9r\" (UniqueName: \"kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r\") pod \"payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:40.040265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.040236    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:40.093589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.093553    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvc9r\" (UniqueName: \"kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r\") pod \"payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:40.236015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.235984    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:40.415957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.415844    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:40.415957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.415898    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:40.607601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:40.607474    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"]
Nov 03 00:35:40.609423 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:35:40.608919    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe6388a_55ba_46d7_8c62_4f077991e60a.slice/crio-d3b2c73c73eb6411ad037d095cb32fd3380e4790808a8a13b61d42749842958b WatchSource:0}: Error finding container d3b2c73c73eb6411ad037d095cb32fd3380e4790808a8a13b61d42749842958b: Status 404 returned error can't find the container with id d3b2c73c73eb6411ad037d095cb32fd3380e4790808a8a13b61d42749842958b
Nov 03 00:35:41.030979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.030937    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" event={"ID":"dbe6388a-55ba-46d7-8c62-4f077991e60a","Type":"ContainerStarted","Data":"06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808"}
Nov 03 00:35:41.031160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.031003    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" event={"ID":"dbe6388a-55ba-46d7-8c62-4f077991e60a","Type":"ContainerStarted","Data":"d3b2c73c73eb6411ad037d095cb32fd3380e4790808a8a13b61d42749842958b"}
Nov 03 00:35:41.031160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.031029    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:41.078455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.073923    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" podStartSLOduration=2.073902937 podStartE2EDuration="2.073902937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:41.072929458 +0000 UTC m=+1823.394466260" watchObservedRunningTime="2026-11-03 00:35:41.073902937 +0000 UTC m=+1823.395439737"
Nov 03 00:35:41.183114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.183079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:41.413424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.413388    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:41.413645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:41.413467    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:42.413290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:42.413252    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:42.413714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:42.413305    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:43.203388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:43.203337    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:43.289894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:43.289855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:43.379491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:43.379456    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:43.413679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:43.413642    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 00:35:43.414159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:43.413706    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 00:35:44.463001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:44.462969    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:44.465516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:44.465494    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:44.475915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:44.475883    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:35:45.455915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:45.455875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:45.556477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:45.556411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:45.936938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:45.936897    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:46.416745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.416662    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"]
Nov 03 00:35:46.423879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.423854    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.467202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.467108    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"]
Nov 03 00:35:46.492486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.481344    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2858fb-openshift-default-dockercfg-97wbn\""
Nov 03 00:35:46.492889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.482279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:46.598314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.598256    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.602544    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.602624    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twptm\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.602721    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.602787    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.602809    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.602909    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.603064    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.603092    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.733536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.714276    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.784107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.784073    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.784487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.784461    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twptm\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.788453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.784723    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.788780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.788756    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.788933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.788913    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.802753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.802718    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.825655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.805466    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.828322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.820001    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.828607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.826045    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.828850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.828830    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.840596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.827614    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.841104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.828157    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.841292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.826486    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.841453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.829749    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.881472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.871808    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.896576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.896494    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.897567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.897544    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twptm\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm\") pod \"e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") " pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:46.909793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:46.909596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:47.043516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.042685    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:47.530635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.530575    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:47.658778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.572676    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:47.658778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.622806    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Nov 03 00:35:47.658778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.657850    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:47.762958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.762558    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 00:35:47.840464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.763999    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:47.840464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:47.827749    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:48.007469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:48.006160    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"]
Nov 03 00:35:48.029374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:48.014686    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:48.029374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:48.014782    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:48.029374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:48.014829    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:35:48.159821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:48.159706    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" event={"ID":"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217","Type":"ContainerStarted","Data":"fa64e6be7d0ae0ad317d252e6a5dba54e24ceb69c87dd9d59956b4cbf61a6d51"}
Nov 03 00:35:48.283507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:48.282110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:49.172632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:49.172583    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" event={"ID":"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217","Type":"ContainerStarted","Data":"618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"}
Nov 03 00:35:49.229570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:49.229520    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podStartSLOduration=3.229504582 podStartE2EDuration="3.229504582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:49.227631624 +0000 UTC m=+1831.549168423" watchObservedRunningTime="2026-11-03 00:35:49.229504582 +0000 UTC m=+1831.551041382"
Nov 03 00:35:50.044089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:50.044043    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:50.049592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:50.049561    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:50.180478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:50.180415    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:50.181323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:50.181305    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:35:50.360580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:50.360489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:50.770132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:50.770090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:51.219423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:51.219389    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:51.626881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:51.626791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:51.746201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:51.746166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:51.935721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:51.935686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:52.069058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:52.069030    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 00:35:52.449071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:52.449024    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:52.738364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:52.738267    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:53.393087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:53.393052    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:53.488265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:53.488233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:54.617834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:54.617800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:54.744230 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:35:54.744189    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:35:55.709749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:55.709707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:55.797844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:55.797816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:55.928311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:55.928278    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:56.039835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:56.039753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:56.608680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:56.608649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:56.714424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:56.714387    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:56.951727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:56.951698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:57.685644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:57.685611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:57.777196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:57.777159    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:35:58.765706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:35:58.765672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:00.494743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:00.494649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:00.582686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:00.582651    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:00.685453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:00.684599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:02.464814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:02.464776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:04.376721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:04.376680    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:06.197199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.197074    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"]
Nov 03 00:36:06.203915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.203890    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.218129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.218100    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"]
Nov 03 00:36:06.225534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.224532    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:36:06.225534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.224582    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wclrs\""
Nov 03 00:36:06.225534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.225163    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:36:06.225534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.225471    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:06.225534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.224460    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:36:06.294820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.294775    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.295005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.294863    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.295132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.295114    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.295227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.295212    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.295325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.295310    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktcp8\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.295395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.295379    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.295453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.295419    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397128    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktcp8\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397170    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397196    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397247    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397290    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397361    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.397445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.397416    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.399616 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.399591    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.405154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.405129    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.405767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.405744    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.407367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.407347    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.409164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.409141    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.422004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.421983    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.422140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.422118    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktcp8\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-l2q2c\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.522462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.522355    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:06.895275 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:36:06.895237    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf177d44a_571f_415d_9328_4936506943f5.slice/crio-5bacc85cb9bf76e24167dda0e6414c6797f9c2858104b7c589a6ab36db4f83d7 WatchSource:0}: Error finding container 5bacc85cb9bf76e24167dda0e6414c6797f9c2858104b7c589a6ab36db4f83d7: Status 404 returned error can't find the container with id 5bacc85cb9bf76e24167dda0e6414c6797f9c2858104b7c589a6ab36db4f83d7
Nov 03 00:36:06.898189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.898154    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:06.898294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.898220    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:06.898294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:06.898271    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"]
Nov 03 00:36:07.310346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.310299    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" event={"ID":"f177d44a-571f-415d-9328-4936506943f5","Type":"ContainerStarted","Data":"2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac"}
Nov 03 00:36:07.310848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.310363    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" event={"ID":"f177d44a-571f-415d-9328-4936506943f5","Type":"ContainerStarted","Data":"5bacc85cb9bf76e24167dda0e6414c6797f9c2858104b7c589a6ab36db4f83d7"}
Nov 03 00:36:07.310848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.310402    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:07.312607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.312582    4550 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-l2q2c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:36:07.312692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.312630    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" podUID="f177d44a-571f-415d-9328-4936506943f5" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:36:07.371747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.371683    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" podStartSLOduration=1.371663755 podStartE2EDuration="1.371663755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:07.369708772 +0000 UTC m=+1849.691245597" watchObservedRunningTime="2026-11-03 00:36:07.371663755 +0000 UTC m=+1849.693200556"
Nov 03 00:36:07.948796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:07.948755    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:08.317544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:08.317459    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:36:09.741418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:09.741386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:10.971947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:10.971912    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:11.268176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:11.268079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:11.440053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:11.440019    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:13.082540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:13.082474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:13.174020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:13.173980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:13.280338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:13.280305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:14.899715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:14.899683    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:15.012524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.012407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:15.383524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.381840    4550 generic.go:356] "Generic (PLEG): container finished" podID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerID="b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7" exitCode=0
Nov 03 00:36:15.383524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.381889    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" event={"ID":"c3204b61-751e-49c9-bcdb-007c8f5206f4","Type":"ContainerDied","Data":"b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"}
Nov 03 00:36:15.383524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.382363    4550 scope.go:122] "RemoveContainer" containerID="b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"
Nov 03 00:36:15.384211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.384162    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:15.384286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.384249    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:15.384351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:15.384298    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:16.410450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:16.410326    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" event={"ID":"c3204b61-751e-49c9-bcdb-007c8f5206f4","Type":"ContainerStarted","Data":"e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9"}
Nov 03 00:36:16.729538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:16.728958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:17.060508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:17.058184    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:36:17.060508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:17.058621    4550 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:36:17.060508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:17.058888    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 00:36:18.060458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:18.060104    4550 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:36:18.060458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:18.060164    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 00:36:18.403449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:18.403152    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:18.496479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:18.495208    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:18.621453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:18.612467    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:18.838469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:18.838379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:19.063456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:19.059848    4550 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:36:19.063456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:19.059917    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 00:36:20.061453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:20.060821    4550 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:36:20.061453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:20.060887    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 00:36:20.514343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:20.514301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:21.060917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.060879    4550 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body=
Nov 03 00:36:21.061199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.061175    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused"
Nov 03 00:36:21.093511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.093472    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"]
Nov 03 00:36:21.093971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.093940    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy" containerID="cri-o://6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046" gracePeriod=30
Nov 03 00:36:21.111797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.111758    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"]
Nov 03 00:36:21.112276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.112206    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" containerID="cri-o://6a2aa14226a8c3c4a238aa297f10a999a432eec88ea9a9727547d413b12456d6" gracePeriod=30
Nov 03 00:36:21.237083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.236996    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-lnm9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:36:21.237338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.237306    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:36:21.385507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.384906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:21.478634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:21.478587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:22.121741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:22.121707    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:36:22.122400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:22.122374    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:36:22.123928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:22.123910    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:36:22.291526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:22.291484    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:23.819471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:23.804049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:24.045064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:24.045030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:24.727109 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:36:24.727073    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:36:25.736424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:25.736384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:26.443754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.443722    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:36:26.551254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.551218    4550 generic.go:356] "Generic (PLEG): container finished" podID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerID="6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046" exitCode=0
Nov 03 00:36:26.551613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.551588    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" event={"ID":"b608fd30-9154-443f-9d44-33dc1be9f0e5","Type":"ContainerDied","Data":"6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046"}
Nov 03 00:36:26.551792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.551773    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll" event={"ID":"b608fd30-9154-443f-9d44-33dc1be9f0e5","Type":"ContainerDied","Data":"69c335bfaf5980168adf591e7d636972aee8df7f456afadacc7e550e18d33afa"}
Nov 03 00:36:26.551923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.551911    4550 scope.go:122] "RemoveContainer" containerID="6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046"
Nov 03 00:36:26.552166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.552154    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"
Nov 03 00:36:26.558728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.558702    4550 generic.go:356] "Generic (PLEG): container finished" podID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerID="6a2aa14226a8c3c4a238aa297f10a999a432eec88ea9a9727547d413b12456d6" exitCode=0
Nov 03 00:36:26.558909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.558892    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" event={"ID":"c8f0e546-9de9-4332-830d-c6e742938d4e","Type":"ContainerDied","Data":"6a2aa14226a8c3c4a238aa297f10a999a432eec88ea9a9727547d413b12456d6"}
Nov 03 00:36:26.566953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.566920    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.571459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.567170    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.571768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.571742    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.571932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.571917    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.572062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.572047    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.572226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.572209    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.576456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.572418    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.576748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.576724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.576996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.576976    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz\") pod \"b608fd30-9154-443f-9d44-33dc1be9f0e5\" (UID: \"b608fd30-9154-443f-9d44-33dc1be9f0e5\") "
Nov 03 00:36:26.592370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.592336    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.597498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.594235    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.597498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.594553    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:36:26.597498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.594762    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.597498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.596877    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.608695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.608651    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.610632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.610585    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "kube-api-access-gt7gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:36:26.615012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.613878    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:36:26.618931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.618673    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token" pod "b608fd30-9154-443f-9d44-33dc1be9f0e5" (UID: "b608fd30-9154-443f-9d44-33dc1be9f0e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:36:26.623622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.623603    4550 scope.go:122] "RemoveContainer" containerID="6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046"
Nov 03 00:36:26.623955 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:36:26.623930    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046\": container with ID starting with 6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046 not found: ID does not exist" containerID="6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046"
Nov 03 00:36:26.624050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.623966    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046"} err="failed to get container status \"6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046\": rpc error: code = NotFound desc = could not find container \"6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046\": container with ID starting with 6bb8c6965e0a0bd795374667eb22f91fa6c1ec1e2b36f6e73a4d3691495f2046 not found: ID does not exist"
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691070    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691112    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7gz\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-kube-api-access-gt7gz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691128    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691142    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691156    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691169    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691181    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b608fd30-9154-443f-9d44-33dc1be9f0e5-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691193    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b608fd30-9154-443f-9d44-33dc1be9f0e5-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.691293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.691206    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b608fd30-9154-443f-9d44-33dc1be9f0e5-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.842181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.842156    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:36:26.974578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.961038    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"]
Nov 03 00:36:26.983482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.983421    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-7nrll"]
Nov 03 00:36:26.996144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996112    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996192    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996254    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996274    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996303    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996511    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996622    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996699    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996721    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.996761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996736    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.996929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996810    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9\") pod \"c8f0e546-9de9-4332-830d-c6e742938d4e\" (UID: \"c8f0e546-9de9-4332-830d-c6e742938d4e\") "
Nov 03 00:36:26.996929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.996899    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.997036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.997000    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.997269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.997245    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:36:26.997709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.997620    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.997709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.997647    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.997709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.997660    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f0e546-9de9-4332-830d-c6e742938d4e-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.999087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.999064    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.999185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.999086    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "kube-api-access-nvbm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:36:26.999185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.999095    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:26.999418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.999392    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:36:26.999629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.999601    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:36:26.999777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:26.999759    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token" pod "c8f0e546-9de9-4332-830d-c6e742938d4e" (UID: "c8f0e546-9de9-4332-830d-c6e742938d4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:36:27.100066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.100029    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbm9\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-kube-api-access-nvbm9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:27.100066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.100064    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:27.100066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.100073    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:27.100288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.100082    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f0e546-9de9-4332-830d-c6e742938d4e-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:36:27.571453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.571061    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"
Nov 03 00:36:27.571453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.571224    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s" event={"ID":"c8f0e546-9de9-4332-830d-c6e742938d4e","Type":"ContainerDied","Data":"8d72558cbf3a24dbe75926c92b575f492bbbba110a2dde15371afaaf8e395948"}
Nov 03 00:36:27.571453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.571324    4550 scope.go:122] "RemoveContainer" containerID="6a2aa14226a8c3c4a238aa297f10a999a432eec88ea9a9727547d413b12456d6"
Nov 03 00:36:27.630588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.630556    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"]
Nov 03 00:36:27.636993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.636957    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-lnm9s"]
Nov 03 00:36:27.700383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.700323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:27.811732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.811676    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" path="/var/lib/kubelet/pods/b608fd30-9154-443f-9d44-33dc1be9f0e5/volumes"
Nov 03 00:36:27.812210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:27.812193    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" path="/var/lib/kubelet/pods/c8f0e546-9de9-4332-830d-c6e742938d4e/volumes"
Nov 03 00:36:28.754393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:28.754357    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:29.318562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:29.318527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:29.419633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:29.419599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:29.787319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:29.787278    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:30.266957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:30.266891    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:30.478454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:30.469235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:31.255554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.255461    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:31.626557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.626519    4550 generic.go:356] "Generic (PLEG): container finished" podID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerID="b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a" exitCode=0
Nov 03 00:36:31.626758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.626599    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" event={"ID":"6c6f197d-8e10-4bee-a8a0-748069364c01","Type":"ContainerDied","Data":"b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"}
Nov 03 00:36:31.627191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.627173    4550 scope.go:122] "RemoveContainer" containerID="b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"
Nov 03 00:36:31.628915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.628885    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:31.629048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.628971    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:31.629048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.629022    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:31.899691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:31.899524    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:36:32.634960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:32.634921    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" event={"ID":"6c6f197d-8e10-4bee-a8a0-748069364c01","Type":"ContainerStarted","Data":"ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f"}
Nov 03 00:36:32.745319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:32.745285    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:36:32.746260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:32.746238    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 00:36:32.746379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:32.746286    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 00:36:32.935027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:32.934997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:33.745614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:33.745583    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 00:36:33.746013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:33.745647    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 00:36:34.436862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:34.436825    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:34.586716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:34.582350    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:34.745094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:34.745009    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 00:36:34.745094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:34.745062    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 00:36:34.752375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:34.752346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:35.746453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:35.745487    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 00:36:35.746453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:35.745572    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 00:36:36.252201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:36.252172    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:36.382454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:36.382052    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:36.787455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:36.784922    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 00:36:36.787455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:36.785003    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 00:36:37.183985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:37.183948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:37.745284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:37.745250    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Nov 03 00:36:37.745776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:37.745306    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Nov 03 00:36:38.750604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:38.750573    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:36:38.751107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:38.751068    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:36:38.751724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:38.751702    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 00:36:39.279775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:39.279724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:40.949661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:40.949623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:42.577453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:42.577299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:42.885836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:42.885743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:44.585230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:44.585196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:44.706009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:44.705974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:46.551516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:46.551404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:47.910275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:47.910243    4550 generic.go:356] "Generic (PLEG): container finished" podID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerID="618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe" exitCode=0
Nov 03 00:36:47.910727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:47.910326    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" event={"ID":"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217","Type":"ContainerDied","Data":"618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"}
Nov 03 00:36:47.910827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:47.910808    4550 scope.go:122] "RemoveContainer" containerID="618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"
Nov 03 00:36:47.912393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:47.912366    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:47.912478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:47.912449    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:47.912535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:47.912485    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:36:48.453268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:48.453228    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:48.921995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:48.921882    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" event={"ID":"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217","Type":"ContainerStarted","Data":"d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3"}
Nov 03 00:36:49.044788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:49.044754    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:36:49.046195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:49.046167    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 00:36:49.046363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:49.046225    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 00:36:50.044188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:50.044153    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 00:36:50.044658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:50.044229    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 00:36:50.147041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:50.146998    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:50.186345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:50.186306    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:36:51.044338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:51.044303    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 00:36:51.044888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:51.044363    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 00:36:51.765455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:51.759945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:51.988085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:51.988051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:52.045025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:52.044940    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 00:36:52.045025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:52.044999    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 00:36:52.992685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:52.992641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:53.059000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:53.058966    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 00:36:53.059760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:53.059727    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 00:36:53.454068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:53.454012    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:53.751264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:53.751177    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:54.045245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:54.045151    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Nov 03 00:36:54.045245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:54.045228    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Nov 03 00:36:54.728604 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:36:54.728564    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:36:55.049320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:55.049243    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:36:55.050451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:55.050412    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 00:36:55.370116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:55.370028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:55.560328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:55.560292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:57.591857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:57.591828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:58.463781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:58.463746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:36:59.212994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:36:59.212961    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:00.900972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:00.900937    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:00.989167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:00.989127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:01.094802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:01.094770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:02.735395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:02.735359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:04.497800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:04.497753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:06.150127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:06.150093    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:08.165666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:08.165635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:09.914877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:09.914841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:11.075256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:11.075215    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:11.175565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:11.175528    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:11.608459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:11.608397    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:13.273217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:13.273171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:14.798754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.798719    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"]
Nov 03 00:37:14.799285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.799273    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy"
Nov 03 00:37:14.799285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.799286    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy"
Nov 03 00:37:14.799386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.799319    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy"
Nov 03 00:37:14.799386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.799325    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy"
Nov 03 00:37:14.800046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.799444    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b608fd30-9154-443f-9d44-33dc1be9f0e5" containerName="istio-proxy"
Nov 03 00:37:14.800046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.799462    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f0e546-9de9-4332-830d-c6e742938d4e" containerName="istio-proxy"
Nov 03 00:37:14.803577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.803547    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.826586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.826542    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"]
Nov 03 00:37:14.839754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.834903    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e62935\"/\"openshift-service-ca.crt\""
Nov 03 00:37:14.839754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.834908    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-e62935\"/\"kube-root-ca.crt\""
Nov 03 00:37:14.839754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.835019    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e62935\"/\"default-dockercfg-897mw\""
Nov 03 00:37:14.840739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.840716    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-e62935\"/\"auth-test-model-e62935-kserve-self-signed-certs\""
Nov 03 00:37:14.852601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.852574    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:14.942955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.942913    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jsqh\" (UniqueName: \"kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.943381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.943343    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.943521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.943416    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.943521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.943478    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.943521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.943517    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.943700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.943646    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:14.960783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:14.960749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.045520    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jsqh\" (UniqueName: \"kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.045667    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.045710    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.045748    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.045784    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.045945    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.047454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.046946    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.056497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.048746    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.060496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.057832    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.067450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.062486    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.071717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.071052    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.076492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.076467    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jsqh\" (UniqueName: \"kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh\") pod \"auth-test-model-e62935-kserve-d88cf4789-246pv\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") " pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.117408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.117374    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:15.487651 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:37:15.487617    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76d881f1_4886_42af_924e_8168316a1788.slice/crio-005ce7e0496ed65473bd618b8e80f482c25129806ff3473d646f40946b1e29aa WatchSource:0}: Error finding container 005ce7e0496ed65473bd618b8e80f482c25129806ff3473d646f40946b1e29aa: Status 404 returned error can't find the container with id 005ce7e0496ed65473bd618b8e80f482c25129806ff3473d646f40946b1e29aa
Nov 03 00:37:15.489949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:15.489924    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"]
Nov 03 00:37:16.173123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:16.173082    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv" event={"ID":"76d881f1-4886-42af-924e-8168316a1788","Type":"ContainerStarted","Data":"ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460"}
Nov 03 00:37:16.173563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:16.173135    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv" event={"ID":"76d881f1-4886-42af-924e-8168316a1788","Type":"ContainerStarted","Data":"005ce7e0496ed65473bd618b8e80f482c25129806ff3473d646f40946b1e29aa"}
Nov 03 00:37:16.214676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:16.214625    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv" podStartSLOduration=2.214610129 podStartE2EDuration="2.214610129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:16.212353561 +0000 UTC m=+1918.533890360" watchObservedRunningTime="2026-11-03 00:37:16.214610129 +0000 UTC m=+1918.536146928"
Nov 03 00:37:16.738307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:16.738271    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:17.118004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:17.117924    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:18.423792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:18.423758    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:18.709901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:18.709806    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:20.225854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:20.225811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:21.815898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:21.815859    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:23.478396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:23.478355    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:24.723873 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:37:24.723833    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:37:25.094657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:25.094563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:25.118581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:25.118543    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:25.131474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:25.131443    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:25.260050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:25.260022    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:37:26.747318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:26.747282    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:28.030453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:28.030391    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"]
Nov 03 00:37:28.043406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:28.043377    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7648e554-f98jp"]
Nov 03 00:37:28.556511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:28.556479    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:29.809135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:29.809039    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a24fad59-6017-48f0-8d43-8056174747a1" path="/var/lib/kubelet/pods/a24fad59-6017-48f0-8d43-8056174747a1/volumes"
Nov 03 00:37:30.417774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:30.417724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:30.518651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:30.518612    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:31.259040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.258999    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"]
Nov 03 00:37:31.264797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.264772    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.304127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.304089    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"]
Nov 03 00:37:31.343947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.343912    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-359f71-openshift-default-dockercfg-spn84\""
Nov 03 00:37:31.405560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.405526    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.405751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.405574    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.405751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.405643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.405855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.405774    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.405855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.405812    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.405958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.405944    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.406051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.406037    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.406106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.406063    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw87x\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.406233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.406194    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.507759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507719    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.507958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507804    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.507958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507823    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw87x\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.507958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507863    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.507958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507934    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.507958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507958    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.507979    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508036    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508264    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508305    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508347    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508493    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508558    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.508753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.508735    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.510378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.510318    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.510593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.510576    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.527844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.527816    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.528023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.528005    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw87x\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.579875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.579650    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:31.901742 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:37:31.901713    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9641d0eb_647f_4294_a396_647398bfaab3.slice/crio-517e83b8294da933c018596ee9f8c76313caadd1048ee4dce25319759a0fe5d3 WatchSource:0}: Error finding container 517e83b8294da933c018596ee9f8c76313caadd1048ee4dce25319759a0fe5d3: Status 404 returned error can't find the container with id 517e83b8294da933c018596ee9f8c76313caadd1048ee4dce25319759a0fe5d3
Nov 03 00:37:31.904222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.904193    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"]
Nov 03 00:37:31.905193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.905152    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:31.905268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.905247    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:31.905306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:31.905288    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:37:32.192465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.183514    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:32.369006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.368958    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" event={"ID":"9641d0eb-647f-4294-a396-647398bfaab3","Type":"ContainerStarted","Data":"62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71"}
Nov 03 00:37:32.369546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.369016    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" event={"ID":"9641d0eb-647f-4294-a396-647398bfaab3","Type":"ContainerStarted","Data":"517e83b8294da933c018596ee9f8c76313caadd1048ee4dce25319759a0fe5d3"}
Nov 03 00:37:32.419985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.419931    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podStartSLOduration=1.419915947 podStartE2EDuration="1.419915947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:32.418173037 +0000 UTC m=+1934.739709860" watchObservedRunningTime="2026-11-03 00:37:32.419915947 +0000 UTC m=+1934.741452799"
Nov 03 00:37:32.586512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.580503    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:32.594459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.593577    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:37:32.594459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:32.593637    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 00:37:33.594516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:33.593925    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:37:33.594516 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:33.593989    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 00:37:34.027902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:34.027864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:34.591507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:34.583768    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:37:34.591507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:34.583831    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 00:37:35.221555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:35.221513    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"]
Nov 03 00:37:35.330914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:35.330880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:35.428798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:35.428606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:35.598693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:35.594592    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:37:35.598693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:35.594661    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 00:37:35.796456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:35.795140    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:36.090455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:36.089857    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:36.582455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:36.581696    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:37:36.582455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:36.581753    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 00:37:37.593456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:37.587323    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:37:37.593456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:37.587377    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 03 00:37:37.809777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:37.809741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:38.658467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:38.658406    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:38.659971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:38.659947    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:38.664395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:38.664366    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 00:37:39.507867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:39.507773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:40.222246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.222200    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh" podUID="189586c0-e6ba-4806-8742-1054cbadf17a" containerName="payload-pre-processing" containerID="cri-o://04c2d1caad0004c610a4a0805db5dd65528df46484c0081927da26874bf8237d" gracePeriod=25
Nov 03 00:37:40.457347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.456479    4550 generic.go:356] "Generic (PLEG): container finished" podID="189586c0-e6ba-4806-8742-1054cbadf17a" containerID="04c2d1caad0004c610a4a0805db5dd65528df46484c0081927da26874bf8237d" exitCode=0
Nov 03 00:37:40.457347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.457280    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh" event={"ID":"189586c0-e6ba-4806-8742-1054cbadf17a","Type":"ContainerDied","Data":"04c2d1caad0004c610a4a0805db5dd65528df46484c0081927da26874bf8237d"}
Nov 03 00:37:40.533131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.533099    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:37:40.623572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.623538    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume\") pod \"189586c0-e6ba-4806-8742-1054cbadf17a\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") "
Nov 03 00:37:40.623783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.623594    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd\" (UniqueName: \"kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd\") pod \"189586c0-e6ba-4806-8742-1054cbadf17a\" (UID: \"189586c0-e6ba-4806-8742-1054cbadf17a\") "
Nov 03 00:37:40.623993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.623969    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume" pod "189586c0-e6ba-4806-8742-1054cbadf17a" (UID: "189586c0-e6ba-4806-8742-1054cbadf17a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:37:40.624118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.624100    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/189586c0-e6ba-4806-8742-1054cbadf17a-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:37:40.626059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.626030    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd" pod "189586c0-e6ba-4806-8742-1054cbadf17a" (UID: "189586c0-e6ba-4806-8742-1054cbadf17a"). InnerVolumeSpecName "kube-api-access-lmfqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:37:40.725290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:40.725247    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmfqd\" (UniqueName: \"kubernetes.io/projected/189586c0-e6ba-4806-8742-1054cbadf17a-kube-api-access-lmfqd\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:37:41.140242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.140209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:41.466598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.466555    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh" event={"ID":"189586c0-e6ba-4806-8742-1054cbadf17a","Type":"ContainerDied","Data":"6a6c3270b17dca009974374316bca3f99a1685b57a3b5be7eae4a6bc431aec20"}
Nov 03 00:37:41.467088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.466570    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"
Nov 03 00:37:41.467088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.466654    4550 scope.go:122] "RemoveContainer" containerID="04c2d1caad0004c610a4a0805db5dd65528df46484c0081927da26874bf8237d"
Nov 03 00:37:41.478869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.478843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:41.522907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.522872    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"]
Nov 03 00:37:41.535863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.535830    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c59124-655b84ff5-bgnzh"]
Nov 03 00:37:41.809961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:41.809879    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="189586c0-e6ba-4806-8742-1054cbadf17a" path="/var/lib/kubelet/pods/189586c0-e6ba-4806-8742-1054cbadf17a/volumes"
Nov 03 00:37:42.327104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:42.327065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:42.937484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:42.937422    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:44.639044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:44.639006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:45.030372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:45.030324    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"]
Nov 03 00:37:45.043492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:45.043459    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e67b4f6-jkfd2"]
Nov 03 00:37:45.808127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:45.808093    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724c8687-d7e4-4c6d-bee0-31a057617c45" path="/var/lib/kubelet/pods/724c8687-d7e4-4c6d-bee0-31a057617c45/volumes"
Nov 03 00:37:46.411194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:46.411151    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:48.128992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:48.128941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:48.567755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:48.567720    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:49.934848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:49.934795    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:51.353205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:51.353174    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:52.154254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:52.154217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:53.778360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:53.778297    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:54.726359 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:37:54.726315    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:37:55.462953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:55.462918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:56.034793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:56.034761    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:56.146399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:56.146364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:56.755618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:56.755573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:57.111866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:57.111777    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"]
Nov 03 00:37:57.112083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:57.112039    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy" containerID="cri-o://e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9" gracePeriod=30
Nov 03 00:37:57.130808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:57.130774    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:57.354508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:57.354472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:57.800308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:57.800270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:58.031233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:58.031193    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"]
Nov 03 00:37:58.044551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:58.044515    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"]
Nov 03 00:37:58.048268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:58.048240    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-63ac1c94-qk8hh"]
Nov 03 00:37:58.051760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:58.051695    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1130e749-rt45r"]
Nov 03 00:37:58.263297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:58.263261    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:58.425290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:58.425259    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:59.490002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:59.489970    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:37:59.810297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:59.810206    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2709b0bb-c4e7-4182-8297-d4b8fe068fa1" path="/var/lib/kubelet/pods/2709b0bb-c4e7-4182-8297-d4b8fe068fa1/volumes"
Nov 03 00:37:59.810772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:37:59.810754    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b8fdd9-039b-46f6-bb7d-aca26fd3624b" path="/var/lib/kubelet/pods/75b8fdd9-039b-46f6-bb7d-aca26fd3624b/volumes"
Nov 03 00:38:01.333723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:01.333689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:01.607873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:01.607784    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:01.842324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:01.842286    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:02.436122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.436096    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:38:02.523192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523160    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523304    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523363    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523423    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523483    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523522    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523578    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523689    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.523795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523734    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy\") pod \"c3204b61-751e-49c9-bcdb-007c8f5206f4\" (UID: \"c3204b61-751e-49c9-bcdb-007c8f5206f4\") "
Nov 03 00:38:02.524228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.523481    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.524288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.524019    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.524288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.524201    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.524288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.524225    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.524665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.524642    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:38:02.540176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.540153    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "kube-api-access-pvx4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:38:02.540300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.540272    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:38:02.540400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.540380    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:38:02.541752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.541726    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy" pod "c3204b61-751e-49c9-bcdb-007c8f5206f4" (UID: "c3204b61-751e-49c9-bcdb-007c8f5206f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.625733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625681    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625732    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvx4d\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-kube-api-access-pvx4d\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625744    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625752    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625761    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3204b61-751e-49c9-bcdb-007c8f5206f4-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625770    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625783    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625795    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3204b61-751e-49c9-bcdb-007c8f5206f4-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.625936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.625803    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3204b61-751e-49c9-bcdb-007c8f5206f4-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.659560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.659526    4550 generic.go:356] "Generic (PLEG): container finished" podID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerID="e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9" exitCode=0
Nov 03 00:38:02.659749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.659596    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"
Nov 03 00:38:02.659749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.659610    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" event={"ID":"c3204b61-751e-49c9-bcdb-007c8f5206f4","Type":"ContainerDied","Data":"e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9"}
Nov 03 00:38:02.659749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.659675    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc" event={"ID":"c3204b61-751e-49c9-bcdb-007c8f5206f4","Type":"ContainerDied","Data":"fd9aa460f3d79fbc7986f5d55358739dd7423ff10eaffd8519337a5e9c669891"}
Nov 03 00:38:02.659749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.659707    4550 scope.go:122] "RemoveContainer" containerID="e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9"
Nov 03 00:38:02.672474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.672454    4550 scope.go:122] "RemoveContainer" containerID="b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"
Nov 03 00:38:02.682482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.682460    4550 scope.go:122] "RemoveContainer" containerID="e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9"
Nov 03 00:38:02.682762 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:38:02.682742    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9\": container with ID starting with e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9 not found: ID does not exist" containerID="e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9"
Nov 03 00:38:02.682828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.682776    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9"} err="failed to get container status \"e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9\": rpc error: code = NotFound desc = could not find container \"e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9\": container with ID starting with e4f69ea1fc18f84f97712b189dd53f3e9c8918b5f09f39fe29031daaf74e61a9 not found: ID does not exist"
Nov 03 00:38:02.682828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.682800    4550 scope.go:122] "RemoveContainer" containerID="b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"
Nov 03 00:38:02.683140 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:38:02.683113    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7\": container with ID starting with b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7 not found: ID does not exist" containerID="b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"
Nov 03 00:38:02.683264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.683145    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7"} err="failed to get container status \"b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7\": rpc error: code = NotFound desc = could not find container \"b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7\": container with ID starting with b1c8448d255ff6f06f9f3537bb1ca405eedc32233fcc2082749100d0ed02f0c7 not found: ID does not exist"
Nov 03 00:38:02.722103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.722070    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"]
Nov 03 00:38:02.728278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:02.728245    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-cq9rc"]
Nov 03 00:38:03.435556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.435523    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:03.548794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.548760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:03.606443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.606404    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"]
Nov 03 00:38:03.607223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607201    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy"
Nov 03 00:38:03.607223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607223    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy"
Nov 03 00:38:03.607368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607239    4550 cpu_manager.go:414] "containerMap: removing container" podUID="189586c0-e6ba-4806-8742-1054cbadf17a" containerName="payload-pre-processing"
Nov 03 00:38:03.607368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607248    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="189586c0-e6ba-4806-8742-1054cbadf17a" containerName="payload-pre-processing"
Nov 03 00:38:03.607368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607259    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy"
Nov 03 00:38:03.607368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607267    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy"
Nov 03 00:38:03.607553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607399    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy"
Nov 03 00:38:03.607553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607412    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="189586c0-e6ba-4806-8742-1054cbadf17a" containerName="payload-pre-processing"
Nov 03 00:38:03.607553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.607420    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" containerName="istio-proxy"
Nov 03 00:38:03.613095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.613077    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.629112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.629086    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"]
Nov 03 00:38:03.637849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.637819    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-b43332\""
Nov 03 00:38:03.637987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.637912    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-b43332-dockercfg-f7dlq\""
Nov 03 00:38:03.643658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.643629    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6\") pod \"payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.643810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.643671    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.749504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.748548    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6\") pod \"payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.749504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.748619    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.758866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.749837    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.813488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.813129    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6\") pod \"payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.935546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.934735    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3204b61-751e-49c9-bcdb-007c8f5206f4" path="/var/lib/kubelet/pods/c3204b61-751e-49c9-bcdb-007c8f5206f4/volumes"
Nov 03 00:38:03.935546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.935234    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"]
Nov 03 00:38:03.935546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.935269    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:03.954661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.954628    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"]
Nov 03 00:38:03.959711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:03.956570    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.036692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.036662    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"]
Nov 03 00:38:04.041581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.041550    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-307a4272-6fwv5"]
Nov 03 00:38:04.075638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.075605    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") " pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.075969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.075943    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x6tn\" (UniqueName: \"kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn\") pod \"payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") " pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.178041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.178003    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") " pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.178247 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.178079    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x6tn\" (UniqueName: \"kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn\") pod \"payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") " pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.179133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.179093    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume\") pod \"payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") " pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.204788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.204753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x6tn\" (UniqueName: \"kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn\") pod \"payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") " pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.295111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.294702    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"]
Nov 03 00:38:04.310886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.310848    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:04.689798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.689766    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"]
Nov 03 00:38:04.698107 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:38:04.698078    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c45ffe_6f52_47d9_9cdf_bccc2b482da1.slice/crio-c7d29486731d129ff840878623a19db8e17867b32fde6f11fda16a931a2f4523 WatchSource:0}: Error finding container c7d29486731d129ff840878623a19db8e17867b32fde6f11fda16a931a2f4523: Status 404 returned error can't find the container with id c7d29486731d129ff840878623a19db8e17867b32fde6f11fda16a931a2f4523
Nov 03 00:38:04.761833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.761798    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt" event={"ID":"60c45ffe-6f52-47d9-9cdf-bccc2b482da1","Type":"ContainerStarted","Data":"c7d29486731d129ff840878623a19db8e17867b32fde6f11fda16a931a2f4523"}
Nov 03 00:38:04.764310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.763593    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq" event={"ID":"78f291f2-be47-49b0-b624-7d2b4808c7ee","Type":"ContainerStarted","Data":"84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a"}
Nov 03 00:38:04.764310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.763653    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq" event={"ID":"78f291f2-be47-49b0-b624-7d2b4808c7ee","Type":"ContainerStarted","Data":"038b7c2b306bbfd308934fb0e7855d9a058939e36541efdf617799d2acb776d1"}
Nov 03 00:38:04.764310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.763838    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:04.809266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:04.809202    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq" podStartSLOduration=1.809184679 podStartE2EDuration="1.809184679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:04.803993249 +0000 UTC m=+1967.125530050" watchObservedRunningTime="2026-11-03 00:38:04.809184679 +0000 UTC m=+1967.130721487"
Nov 03 00:38:05.248894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:05.248858    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:05.772184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:05.772145    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt" event={"ID":"60c45ffe-6f52-47d9-9cdf-bccc2b482da1","Type":"ContainerStarted","Data":"e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17"}
Nov 03 00:38:05.772688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:05.772393    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:05.808860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:05.808832    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493" path="/var/lib/kubelet/pods/40cb72e3-e1db-4ca2-9ff7-bd1ecfdec493/volumes"
Nov 03 00:38:05.814930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:05.814869    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt" podStartSLOduration=2.814856248 podStartE2EDuration="2.814856248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:05.811319098 +0000 UTC m=+1968.132855898" watchObservedRunningTime="2026-11-03 00:38:05.814856248 +0000 UTC m=+1968.136393053"
Nov 03 00:38:06.964498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:06.964457    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:08.602523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:08.602491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:08.713262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:08.713219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:10.453233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:10.453179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:11.196013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:11.195979    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:12.028159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:12.028123    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"]
Nov 03 00:38:12.044365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:12.044328    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4439f591-wcqjt"]
Nov 03 00:38:12.125111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:12.125070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:13.809884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:13.809838    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397dd2de-fdea-4c0c-b8e1-3a651046d0ae" path="/var/lib/kubelet/pods/397dd2de-fdea-4c0c-b8e1-3a651046d0ae/volumes"
Nov 03 00:38:13.835064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:13.835029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:14.035213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:14.035175    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"]
Nov 03 00:38:14.045843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:14.045818    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ade797b8-j8nvj"]
Nov 03 00:38:15.579132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:15.579098    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:15.775383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:15.775354    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:38:15.810284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:15.810247    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca8f8e3-eb8c-482c-94cd-086908a3961e" path="/var/lib/kubelet/pods/2ca8f8e3-eb8c-482c-94cd-086908a3961e/volumes"
Nov 03 00:38:16.787012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:16.786985    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:38:17.262940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:17.262907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:19.013710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:19.013674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:20.728138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:20.728100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:21.103912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.103823    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"]
Nov 03 00:38:21.109786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.109741    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.130900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.130869    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"]
Nov 03 00:38:21.137341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.137309    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"openshift-service-ca.crt\""
Nov 03 00:38:21.137341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.137351    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"kube-root-ca.crt\""
Nov 03 00:38:21.139140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.139115    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"default-dockercfg-d8kkj\""
Nov 03 00:38:21.139540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.139142    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"auth-test-model-20109d-kserve-self-signed-certs\""
Nov 03 00:38:21.169178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.169143    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:21.242522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.242477    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.242709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.242598    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.242760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.242706    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.242817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.242731    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.242908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.242879    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.243060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.242957    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8nt9\" (UniqueName: \"kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.346245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.345150    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.346245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.345321    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.346245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.345367    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.346245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.345458    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.346245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.345573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8nt9\" (UniqueName: \"kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.346245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.345644    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.348597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.347132    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.349096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.349071    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.350241 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.350218    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.405049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.405014    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8nt9\" (UniqueName: \"kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.429505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.411156    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.429505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.416842    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm\") pod \"auth-test-model-20109d-kserve-bbfb4869-bssjp\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") " pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.438306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.438206    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:21.816476 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:38:21.816421    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cdd1bb1_e415_47fc_94cd_f49d60726c25.slice/crio-6deb75326df54e966a0e3fe690b6101c67d9c9b6a51e9a5fa13b13e5ccd11cff WatchSource:0}: Error finding container 6deb75326df54e966a0e3fe690b6101c67d9c9b6a51e9a5fa13b13e5ccd11cff: Status 404 returned error can't find the container with id 6deb75326df54e966a0e3fe690b6101c67d9c9b6a51e9a5fa13b13e5ccd11cff
Nov 03 00:38:21.820029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.820000    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"]
Nov 03 00:38:21.912233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.912189    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" event={"ID":"4cdd1bb1-e415-47fc-94cd-f49d60726c25","Type":"ContainerStarted","Data":"44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527"}
Nov 03 00:38:21.912425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.912244    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" event={"ID":"4cdd1bb1-e415-47fc-94cd-f49d60726c25","Type":"ContainerStarted","Data":"6deb75326df54e966a0e3fe690b6101c67d9c9b6a51e9a5fa13b13e5ccd11cff"}
Nov 03 00:38:21.957015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:21.956165    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" podStartSLOduration=0.956145883 podStartE2EDuration="956.145883ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:21.953069544 +0000 UTC m=+1984.274606365" watchObservedRunningTime="2026-11-03 00:38:21.956145883 +0000 UTC m=+1984.277682684"
Nov 03 00:38:22.031481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:22.031444    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"]
Nov 03 00:38:22.034880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:22.034855    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27139631-ct6ds"]
Nov 03 00:38:22.439123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:22.439085    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:22.525134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:22.525099    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:23.809856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:23.809820    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96570bfc-c33f-46f4-8ff8-0bfb4987559e" path="/var/lib/kubelet/pods/96570bfc-c33f-46f4-8ff8-0bfb4987559e/volumes"
Nov 03 00:38:24.134787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:24.134689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:24.723805 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:38:24.723773    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:38:25.742372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:25.742335    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:25.841843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:25.841800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:27.029648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:27.029608    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"]
Nov 03 00:38:27.043376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:27.043340    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-805491ef-zqzqc"]
Nov 03 00:38:27.775529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:27.775490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:27.809411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:27.809372    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2887ae57-4d07-4bef-bff4-525c337820a9" path="/var/lib/kubelet/pods/2887ae57-4d07-4bef-bff4-525c337820a9/volumes"
Nov 03 00:38:29.535895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:29.535866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:31.218020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:31.217980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:31.438496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:31.438465    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:31.451029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:31.450997    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:32.039817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:32.039785    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:38:32.918930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:32.918895    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:34.596860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:34.596822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:36.245657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:36.245623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:37.029478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:37.029445    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"]
Nov 03 00:38:37.042094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:37.042059    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c2dd8f-qj2g6"]
Nov 03 00:38:37.809549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:37.809515    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2ba285-6b41-4c50-b155-9a7cdf04439a" path="/var/lib/kubelet/pods/0d2ba285-6b41-4c50-b155-9a7cdf04439a/volumes"
Nov 03 00:38:37.859539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:37.859467    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:39.631832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:39.631792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:41.326344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:41.326302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:43.017850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:43.017816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:43.029698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:43.029666    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"]
Nov 03 00:38:43.042725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:43.042700    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85622cd9-p97mp"]
Nov 03 00:38:43.811814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:43.811769    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a10d957-479e-48db-a76e-563e509e40b5" path="/var/lib/kubelet/pods/2a10d957-479e-48db-a76e-563e509e40b5/volumes"
Nov 03 00:38:44.804061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:44.804032    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:46.528729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:46.528681    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:48.129685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:48.129644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:49.863194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:49.863152    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:51.987045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:51.987003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:52.505884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.505850    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"]
Nov 03 00:38:52.510339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.510297    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.526552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.526526    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a07a90-openshift-default-dockercfg-ttwzx\""
Nov 03 00:38:52.553506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.550606    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"]
Nov 03 00:38:52.686325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686283    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686345    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686445    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686474    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686499    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686604    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686800    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.686880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.686838    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.788969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.788874    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.788969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.788941    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789021    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789046    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789077    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789176    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789209    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789345    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.789528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789380    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.790361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789804    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.790361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.789830    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.790361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.790088    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.790361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.790301    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.790720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.790367    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.791814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.791791    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.792178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.792157    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.809777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.809742    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.810057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.810038    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4\") pod \"e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") " pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:52.829537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:52.829494    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:53.156479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.156443    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"]
Nov 03 00:38:53.158441 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.158394    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:53.158568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.158491    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:53.158568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.158538    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:38:53.160508 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:38:53.160468    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda825147e_2d97_4a2c_b305_24b2b19c6487.slice/crio-c9cdc1df5e32493de8fa3affdda5137255ea0b66d56bc6da71bfaadea5669cc4 WatchSource:0}: Error finding container c9cdc1df5e32493de8fa3affdda5137255ea0b66d56bc6da71bfaadea5669cc4: Status 404 returned error can't find the container with id c9cdc1df5e32493de8fa3affdda5137255ea0b66d56bc6da71bfaadea5669cc4
Nov 03 00:38:53.238301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.238263    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" event={"ID":"a825147e-2d97-4a2c-b305-24b2b19c6487","Type":"ContainerStarted","Data":"8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505"}
Nov 03 00:38:53.238475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.238333    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" event={"ID":"a825147e-2d97-4a2c-b305-24b2b19c6487","Type":"ContainerStarted","Data":"c9cdc1df5e32493de8fa3affdda5137255ea0b66d56bc6da71bfaadea5669cc4"}
Nov 03 00:38:53.286145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.286082    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podStartSLOduration=1.286068099 podStartE2EDuration="1.286068099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:53.28378323 +0000 UTC m=+2015.605320031" watchObservedRunningTime="2026-11-03 00:38:53.286068099 +0000 UTC m=+2015.607604899"
Nov 03 00:38:53.829643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.829607    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:38:53.843918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:53.843866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:54.728239 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:38:54.728192    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:38:54.831406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:54.831372    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:38:54.831624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:54.831454    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 00:38:55.726304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:55.726266    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:55.830442 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:55.830221    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:38:55.830963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:55.830512    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 00:38:56.829996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:56.829960    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:38:56.830181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:56.830018    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 00:38:57.410705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:57.410662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:57.591456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:57.587830    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:57.830271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:57.830188    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:38:57.830271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:57.830251    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 00:38:58.028622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:58.028586    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"]
Nov 03 00:38:58.040956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:58.040922    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0c8132-4bm9d"]
Nov 03 00:38:58.847084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:58.845797    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:38:58.847084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:58.845872    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 00:38:58.873454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:58.851852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:59.415777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:59.415745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:38:59.810457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:59.810351    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de9bcd2-da6a-46d0-952c-5daebca38c52" path="/var/lib/kubelet/pods/6de9bcd2-da6a-46d0-952c-5daebca38c52/volumes"
Nov 03 00:38:59.829891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:59.829854    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Nov 03 00:38:59.830078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:38:59.829919    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Nov 03 00:39:00.834566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:00.834535    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:39:00.835086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:00.834857    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:39:00.835697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:00.835680    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 00:39:01.148972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:01.148883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:02.832559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:02.832521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:03.040187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:03.040151    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"]
Nov 03 00:39:03.054228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:03.054190    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53edfea9-rlvmm"]
Nov 03 00:39:03.808449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:03.808406    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf025b5-c19c-40cb-9057-a978ca5e5b64" path="/var/lib/kubelet/pods/ecf025b5-c19c-40cb-9057-a978ca5e5b64/volumes"
Nov 03 00:39:04.507573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:04.507539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:06.194132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:06.194093    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:07.858946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:07.858910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:09.462687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:09.462648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:11.239612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:11.239573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:11.340793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:11.340749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:12.946132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:12.945877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:13.051820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:13.051782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:13.463023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:13.462981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:14.031881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:14.031850    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"]
Nov 03 00:39:14.044812 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:14.044785    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4f06252-djzxc"]
Nov 03 00:39:14.624660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:14.624627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:15.807950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:15.807915    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bc8e04-6db0-4aa4-a641-56e79f2bc159" path="/var/lib/kubelet/pods/a8bc8e04-6db0-4aa4-a641-56e79f2bc159/volumes"
Nov 03 00:39:16.290887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:16.290839    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:17.028276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:17.028238    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"]
Nov 03 00:39:17.041467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:17.041422    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c3d74f-x4str"]
Nov 03 00:39:17.810299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:17.810262    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d63564-456a-47c0-9df0-21a63ab64fc4" path="/var/lib/kubelet/pods/c2d63564-456a-47c0-9df0-21a63ab64fc4/volumes"
Nov 03 00:39:18.033393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.033361    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:18.521532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.521492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:18.890998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.890907    4550 scope.go:122] "RemoveContainer" containerID="bafb85bda8fdeab7b0732083c9b37c58d48ff8a349f535430714ff3115743a16"
Nov 03 00:39:18.902850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.902826    4550 scope.go:122] "RemoveContainer" containerID="841b1e138d61d7754c2b9cbb3b4df9c375238b630343955d101d1d623d8c0f7d"
Nov 03 00:39:18.912048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.912025    4550 scope.go:122] "RemoveContainer" containerID="7fdcfd9a4ca06ce6d0bf8d458813e9bd8a41a07a14c7128f1af1ca983ed80057"
Nov 03 00:39:18.921122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.921100    4550 scope.go:122] "RemoveContainer" containerID="b14b204e697b9846bbd320156f29d7ee6848cd9cbcac0a700b2dae3e2411e000"
Nov 03 00:39:18.929840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.929822    4550 scope.go:122] "RemoveContainer" containerID="52cb5aaf1f0a21c96115cd148eadc1ab6ec5cc5caf7b1036fef48975d9489856"
Nov 03 00:39:18.938390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.938375    4550 scope.go:122] "RemoveContainer" containerID="1297bcb3492b3826222fe21927c1c72468e91ee9b9fcf90f6a2191743f290b0b"
Nov 03 00:39:18.947631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:18.947615    4550 scope.go:122] "RemoveContainer" containerID="7eb265fdd374293f0116b7cc51c3039cb0555192761c4aec01046545d8a3b9ec"
Nov 03 00:39:19.730684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:19.730656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:21.447044 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:21.447005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:23.458457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:23.458407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:24.031668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:24.031627    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"]
Nov 03 00:39:24.044545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:24.044507    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-935eadee-lqpd8"]
Nov 03 00:39:24.727646 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:39:24.727602    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:39:25.216547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:25.216514    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:25.809006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:25.808970    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2715d399-6bbc-47c5-83ff-5349d2f63be0" path="/var/lib/kubelet/pods/2715d399-6bbc-47c5-83ff-5349d2f63be0/volumes"
Nov 03 00:39:26.904536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:26.904501    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:28.580189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:28.580144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:30.335560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:30.335528    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:32.042818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:32.042778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:33.035796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:33.035760    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"]
Nov 03 00:39:33.048477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:33.048449    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-268b17e8-fzlpg"]
Nov 03 00:39:33.727106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:33.727065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:33.813826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:33.813792    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0401864d-b9b3-4dcf-a9ba-9b7fd67258cc" path="/var/lib/kubelet/pods/0401864d-b9b3-4dcf-a9ba-9b7fd67258cc/volumes"
Nov 03 00:39:35.323699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:35.323667    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:36.029802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:36.029767    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"]
Nov 03 00:39:36.042613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:36.042585    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e022b265-kjx2m"]
Nov 03 00:39:37.025864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:37.025828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:37.476488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:37.476452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:37.589305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:37.589255    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:37.811222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:37.811142    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708748e4-f912-4b13-8df0-60f0cf150fcd" path="/var/lib/kubelet/pods/708748e4-f912-4b13-8df0-60f0cf150fcd/volumes"
Nov 03 00:39:38.627186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:38.627143    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:40.446337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:40.446293    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:42.139013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:42.138975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:43.835203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:43.835151    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:45.075332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:45.075302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:45.507226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:45.507193    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:47.190335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:47.190294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:48.796276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:48.796229    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:50.435207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:50.435171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:52.235809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:52.235775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:53.028736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:53.028702    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"]
Nov 03 00:39:53.041359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:53.041328    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdbe7b8e-29nr5"]
Nov 03 00:39:53.809627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:53.809591    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e358839-91a8-4ae2-8485-8036cd75657a" path="/var/lib/kubelet/pods/9e358839-91a8-4ae2-8485-8036cd75657a/volumes"
Nov 03 00:39:53.902301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:53.902269    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:54.726658 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:39:54.726623    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:39:55.637462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:55.637404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:55.738453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:55.738382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:55.889727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:55.889637    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"]
Nov 03 00:39:55.920134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:55.920105    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"]
Nov 03 00:39:56.013964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:56.013903    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:56.145243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:56.145154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:57.473952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:57.473916    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:58.029509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:58.029477    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"]
Nov 03 00:39:58.042746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:58.042721    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f82238-kb74s"]
Nov 03 00:39:59.240868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:59.240823    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:39:59.809995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:39:59.809962    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4852a3f-2afd-4a75-812a-d3bce55a5e31" path="/var/lib/kubelet/pods/f4852a3f-2afd-4a75-812a-d3bce55a5e31/volumes"
Nov 03 00:40:00.890422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:00.890365    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt" podUID="60c45ffe-6f52-47d9-9cdf-bccc2b482da1" containerName="payload-processing" containerID="cri-o://e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17" gracePeriod=25
Nov 03 00:40:00.921146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:00.921113    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq" podUID="78f291f2-be47-49b0-b624-7d2b4808c7ee" containerName="payload-pre-processing" containerID="cri-o://84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a" gracePeriod=25
Nov 03 00:40:00.944036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:00.944008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:01.235058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.235035    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:40:01.255855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.255821    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:01.289915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.289893    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:40:01.370627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.370589    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn\" (UniqueName: \"kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn\") pod \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") "
Nov 03 00:40:01.371423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.371399    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6\") pod \"78f291f2-be47-49b0-b624-7d2b4808c7ee\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") "
Nov 03 00:40:01.374948 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.374920    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume\") pod \"78f291f2-be47-49b0-b624-7d2b4808c7ee\" (UID: \"78f291f2-be47-49b0-b624-7d2b4808c7ee\") "
Nov 03 00:40:01.375155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.375134    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume\") pod \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\" (UID: \"60c45ffe-6f52-47d9-9cdf-bccc2b482da1\") "
Nov 03 00:40:01.376093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.376067    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume" pod "60c45ffe-6f52-47d9-9cdf-bccc2b482da1" (UID: "60c45ffe-6f52-47d9-9cdf-bccc2b482da1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:01.376882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.376858    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume" pod "78f291f2-be47-49b0-b624-7d2b4808c7ee" (UID: "78f291f2-be47-49b0-b624-7d2b4808c7ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:01.378829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.378804    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn" pod "60c45ffe-6f52-47d9-9cdf-bccc2b482da1" (UID: "60c45ffe-6f52-47d9-9cdf-bccc2b482da1"). InnerVolumeSpecName "kube-api-access-8x6tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:01.379753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.379717    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6" pod "78f291f2-be47-49b0-b624-7d2b4808c7ee" (UID: "78f291f2-be47-49b0-b624-7d2b4808c7ee"). InnerVolumeSpecName "kube-api-access-w9xt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:01.477032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.476963    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:01.477032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.476990    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x6tn\" (UniqueName: \"kubernetes.io/projected/60c45ffe-6f52-47d9-9cdf-bccc2b482da1-kube-api-access-8x6tn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:01.477032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.477005    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/78f291f2-be47-49b0-b624-7d2b4808c7ee-kube-api-access-w9xt6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:01.477032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.477018    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78f291f2-be47-49b0-b624-7d2b4808c7ee-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:01.797490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.797392    4550 generic.go:356] "Generic (PLEG): container finished" podID="78f291f2-be47-49b0-b624-7d2b4808c7ee" containerID="84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a" exitCode=0
Nov 03 00:40:01.797490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.797473    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"
Nov 03 00:40:01.797677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.797471    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq" event={"ID":"78f291f2-be47-49b0-b624-7d2b4808c7ee","Type":"ContainerDied","Data":"84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a"}
Nov 03 00:40:01.797677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.797544    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq" event={"ID":"78f291f2-be47-49b0-b624-7d2b4808c7ee","Type":"ContainerDied","Data":"038b7c2b306bbfd308934fb0e7855d9a058939e36541efdf617799d2acb776d1"}
Nov 03 00:40:01.797677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.797568    4550 scope.go:122] "RemoveContainer" containerID="84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a"
Nov 03 00:40:01.799362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.799330    4550 generic.go:356] "Generic (PLEG): container finished" podID="60c45ffe-6f52-47d9-9cdf-bccc2b482da1" containerID="e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17" exitCode=0
Nov 03 00:40:01.799496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.799388    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"
Nov 03 00:40:01.799557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.799392    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt" event={"ID":"60c45ffe-6f52-47d9-9cdf-bccc2b482da1","Type":"ContainerDied","Data":"e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17"}
Nov 03 00:40:01.799621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.799594    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt" event={"ID":"60c45ffe-6f52-47d9-9cdf-bccc2b482da1","Type":"ContainerDied","Data":"c7d29486731d129ff840878623a19db8e17867b32fde6f11fda16a931a2f4523"}
Nov 03 00:40:01.809984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.809967    4550 scope.go:122] "RemoveContainer" containerID="84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a"
Nov 03 00:40:01.810261 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:01.810239    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a\": container with ID starting with 84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a not found: ID does not exist" containerID="84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a"
Nov 03 00:40:01.810364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.810270    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a"} err="failed to get container status \"84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a\": rpc error: code = NotFound desc = could not find container \"84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a\": container with ID starting with 84f1aa1c596eede98c518379afd250e9245fe559e07618de82b34ffa38af860a not found: ID does not exist"
Nov 03 00:40:01.810364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.810292    4550 scope.go:122] "RemoveContainer" containerID="e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17"
Nov 03 00:40:01.820138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.820123    4550 scope.go:122] "RemoveContainer" containerID="e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17"
Nov 03 00:40:01.820386 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:01.820365    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17\": container with ID starting with e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17 not found: ID does not exist" containerID="e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17"
Nov 03 00:40:01.820459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.820392    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17"} err="failed to get container status \"e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17\": rpc error: code = NotFound desc = could not find container \"e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17\": container with ID starting with e4994f66317c0e9ad79ecc7fbf7fd38b1a1b1ff2116354878792df766e65ea17 not found: ID does not exist"
Nov 03 00:40:01.866975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.866952    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"]
Nov 03 00:40:01.879764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.879738    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-b43332-7d5fcfc999-fc6lq"]
Nov 03 00:40:01.915482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.915459    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"]
Nov 03 00:40:01.924467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.924444    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-b43332-5787b6764c-rd7tt"]
Nov 03 00:40:01.950059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950031    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"]
Nov 03 00:40:01.950588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950573    4550 cpu_manager.go:414] "containerMap: removing container" podUID="78f291f2-be47-49b0-b624-7d2b4808c7ee" containerName="payload-pre-processing"
Nov 03 00:40:01.950641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950589    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78f291f2-be47-49b0-b624-7d2b4808c7ee" containerName="payload-pre-processing"
Nov 03 00:40:01.950641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950602    4550 cpu_manager.go:414] "containerMap: removing container" podUID="60c45ffe-6f52-47d9-9cdf-bccc2b482da1" containerName="payload-processing"
Nov 03 00:40:01.950641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950607    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c45ffe-6f52-47d9-9cdf-bccc2b482da1" containerName="payload-processing"
Nov 03 00:40:01.950745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950689    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="78f291f2-be47-49b0-b624-7d2b4808c7ee" containerName="payload-pre-processing"
Nov 03 00:40:01.950745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.950698    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c45ffe-6f52-47d9-9cdf-bccc2b482da1" containerName="payload-processing"
Nov 03 00:40:01.955181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.955165    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:01.971050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.971032    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:40:01.971208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.971141    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:40:01.974058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:01.974037    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"]
Nov 03 00:40:02.086946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.086875    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.087115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.087097    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkwkx\" (UniqueName: \"kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.087164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.087142    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.188888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.188852    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkwkx\" (UniqueName: \"kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.189070 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.188912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.189070 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.189015    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.189883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.189856    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.191394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.191372    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.209258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.209237    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkwkx\" (UniqueName: \"kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx\") pod \"maas-api-revoke-sub-6c2f4596-hg4g9\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.269203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.269173    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:02.574156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.573128    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"]
Nov 03 00:40:02.576747 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:40:02.576715    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5a33ee_80c4_4654_b704_a154afb81ad3.slice/crio-170352fa883c2c89b939a84999a858ae7d8599356713612693878f04e70ba625 WatchSource:0}: Error finding container 170352fa883c2c89b939a84999a858ae7d8599356713612693878f04e70ba625: Status 404 returned error can't find the container with id 170352fa883c2c89b939a84999a858ae7d8599356713612693878f04e70ba625
Nov 03 00:40:02.643139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.643110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:02.809776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:02.809738    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9" event={"ID":"6f5a33ee-80c4-4654-b704-a154afb81ad3","Type":"ContainerStarted","Data":"170352fa883c2c89b939a84999a858ae7d8599356713612693878f04e70ba625"}
Nov 03 00:40:03.808339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:03.808264    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c45ffe-6f52-47d9-9cdf-bccc2b482da1" path="/var/lib/kubelet/pods/60c45ffe-6f52-47d9-9cdf-bccc2b482da1/volumes"
Nov 03 00:40:03.808833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:03.808815    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78f291f2-be47-49b0-b624-7d2b4808c7ee" path="/var/lib/kubelet/pods/78f291f2-be47-49b0-b624-7d2b4808c7ee/volumes"
Nov 03 00:40:03.818519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:03.818495    4550 generic.go:356] "Generic (PLEG): container finished" podID="6f5a33ee-80c4-4654-b704-a154afb81ad3" containerID="3eda3dee47383073b4d0da7f0c475f91f0ab491212c7b39d4ab73ea8927a128e" exitCode=0
Nov 03 00:40:03.818654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:03.818574    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9" event={"ID":"6f5a33ee-80c4-4654-b704-a154afb81ad3","Type":"ContainerDied","Data":"3eda3dee47383073b4d0da7f0c475f91f0ab491212c7b39d4ab73ea8927a128e"}
Nov 03 00:40:04.341105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:04.341065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:04.994624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:04.994598    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:05.030484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.030416    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca\") pod \"6f5a33ee-80c4-4654-b704-a154afb81ad3\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") "
Nov 03 00:40:05.030641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.030516    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx\" (UniqueName: \"kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx\") pod \"6f5a33ee-80c4-4654-b704-a154afb81ad3\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") "
Nov 03 00:40:05.030706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.030638    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca\") pod \"6f5a33ee-80c4-4654-b704-a154afb81ad3\" (UID: \"6f5a33ee-80c4-4654-b704-a154afb81ad3\") "
Nov 03 00:40:05.031096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.031070    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca" pod "6f5a33ee-80c4-4654-b704-a154afb81ad3" (UID: "6f5a33ee-80c4-4654-b704-a154afb81ad3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:05.031240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.031219    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:05.032986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.032963    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca" pod "6f5a33ee-80c4-4654-b704-a154afb81ad3" (UID: "6f5a33ee-80c4-4654-b704-a154afb81ad3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:05.033316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.033298    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx" pod "6f5a33ee-80c4-4654-b704-a154afb81ad3" (UID: "6f5a33ee-80c4-4654-b704-a154afb81ad3"). InnerVolumeSpecName "kube-api-access-dkwkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:05.132719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.132655    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f5a33ee-80c4-4654-b704-a154afb81ad3-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:05.132719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.132681    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkwkx\" (UniqueName: \"kubernetes.io/projected/6f5a33ee-80c4-4654-b704-a154afb81ad3-kube-api-access-dkwkx\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:05.832506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.832474    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9" event={"ID":"6f5a33ee-80c4-4654-b704-a154afb81ad3","Type":"ContainerDied","Data":"170352fa883c2c89b939a84999a858ae7d8599356713612693878f04e70ba625"}
Nov 03 00:40:05.832622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.832522    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170352fa883c2c89b939a84999a858ae7d8599356713612693878f04e70ba625"
Nov 03 00:40:05.832622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:05.832561    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"
Nov 03 00:40:06.021964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:06.021922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:07.716160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:07.716130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:09.104453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:09.103601    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:09.745996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:09.745957    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:11.389693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:11.389648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:11.490865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:11.490831    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:12.372031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.372002    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"]
Nov 03 00:40:12.372538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.372523    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6f5a33ee-80c4-4654-b704-a154afb81ad3" containerName="revoke-keys"
Nov 03 00:40:12.372591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.372539    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5a33ee-80c4-4654-b704-a154afb81ad3" containerName="revoke-keys"
Nov 03 00:40:12.372651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.372642    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5a33ee-80c4-4654-b704-a154afb81ad3" containerName="revoke-keys"
Nov 03 00:40:12.375828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.375810    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.401584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.401554    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"]
Nov 03 00:40:12.410175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.410152    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:40:12.410322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.410197    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:40:12.483734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.483703    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcv2h\" (UniqueName: \"kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.483976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.483956    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.484069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.483994    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.585126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.585093    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.585126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.585127    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.585329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.585179    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcv2h\" (UniqueName: \"kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.585780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.585753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.587578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.587551    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.605591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.605569    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcv2h\" (UniqueName: \"kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h\") pod \"maas-api-revoke-sub-82b8179f-27nwp\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.695408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.695389    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:12.998955 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:40:12.998044    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a710bd_7cfd_4afa_a338_641fb96ca261.slice/crio-4b02eb00a271c78b337a0cc30daeb09a52953031fa3d994c1d764134ca59393d WatchSource:0}: Error finding container 4b02eb00a271c78b337a0cc30daeb09a52953031fa3d994c1d764134ca59393d: Status 404 returned error can't find the container with id 4b02eb00a271c78b337a0cc30daeb09a52953031fa3d994c1d764134ca59393d
Nov 03 00:40:12.998955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:12.998314    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"]
Nov 03 00:40:13.040715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:13.040693    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:13.973967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:13.973935    4550 generic.go:356] "Generic (PLEG): container finished" podID="20a710bd-7cfd-4afa-a338-641fb96ca261" containerID="4e6a89081bdd190e30b7e17082dd28afa24d6fa5d1b138ca1bdce4ef45d3da27" exitCode=0
Nov 03 00:40:13.974322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:13.974021    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp" event={"ID":"20a710bd-7cfd-4afa-a338-641fb96ca261","Type":"ContainerDied","Data":"4e6a89081bdd190e30b7e17082dd28afa24d6fa5d1b138ca1bdce4ef45d3da27"}
Nov 03 00:40:13.974322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:13.974075    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp" event={"ID":"20a710bd-7cfd-4afa-a338-641fb96ca261","Type":"ContainerStarted","Data":"4b02eb00a271c78b337a0cc30daeb09a52953031fa3d994c1d764134ca59393d"}
Nov 03 00:40:14.667507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:14.667471    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:15.157504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.157477    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:15.317676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.317605    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca\") pod \"20a710bd-7cfd-4afa-a338-641fb96ca261\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") "
Nov 03 00:40:15.317820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.317724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca\") pod \"20a710bd-7cfd-4afa-a338-641fb96ca261\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") "
Nov 03 00:40:15.317820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.317772    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h\" (UniqueName: \"kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h\") pod \"20a710bd-7cfd-4afa-a338-641fb96ca261\" (UID: \"20a710bd-7cfd-4afa-a338-641fb96ca261\") "
Nov 03 00:40:15.318049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.318025    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca" pod "20a710bd-7cfd-4afa-a338-641fb96ca261" (UID: "20a710bd-7cfd-4afa-a338-641fb96ca261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:15.318359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.318324    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:15.319696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.319674    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca" pod "20a710bd-7cfd-4afa-a338-641fb96ca261" (UID: "20a710bd-7cfd-4afa-a338-641fb96ca261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:15.319818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.319804    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h" pod "20a710bd-7cfd-4afa-a338-641fb96ca261" (UID: "20a710bd-7cfd-4afa-a338-641fb96ca261"). InnerVolumeSpecName "kube-api-access-qcv2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:15.418987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.418962    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a710bd-7cfd-4afa-a338-641fb96ca261-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:15.418987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.418986    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcv2h\" (UniqueName: \"kubernetes.io/projected/20a710bd-7cfd-4afa-a338-641fb96ca261-kube-api-access-qcv2h\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:15.988675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.988650    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"
Nov 03 00:40:15.988675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.988658    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp" event={"ID":"20a710bd-7cfd-4afa-a338-641fb96ca261","Type":"ContainerDied","Data":"4b02eb00a271c78b337a0cc30daeb09a52953031fa3d994c1d764134ca59393d"}
Nov 03 00:40:15.988873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:15.988713    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b02eb00a271c78b337a0cc30daeb09a52953031fa3d994c1d764134ca59393d"
Nov 03 00:40:16.009014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:16.008991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:16.109765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:16.109676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:16.422882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:16.422848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:16.858393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:16.858287    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:17.346762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:17.346728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:18.560325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:18.560293    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:19.014772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.014741    4550 scope.go:122] "RemoveContainer" containerID="7cb60baaee1fac136facb5d64eb78b4c8841a763835fc02724d6dcd759f4dfec"
Nov 03 00:40:19.025641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.025623    4550 scope.go:122] "RemoveContainer" containerID="201d49a7414929d9347897e64d4d03e72b6fcdf031dc828efabd5496b4432305"
Nov 03 00:40:19.035130 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.035116    4550 scope.go:122] "RemoveContainer" containerID="a840dab72d6693e119ec8b051dcfee8b6069cb8db52488073029b9b840ba8bde"
Nov 03 00:40:19.043500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.043478    4550 scope.go:122] "RemoveContainer" containerID="339d9812c3e69a07a80b5a1417d27ab0abc042fb44beb01cebdd3b165c42a956"
Nov 03 00:40:19.051383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.051366    4550 scope.go:122] "RemoveContainer" containerID="4c4d2d5196f0936ff44e848b7c3f8fc9cdc1a75ecbb1d76e79d0f0800faa5e12"
Nov 03 00:40:19.060734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.060710    4550 scope.go:122] "RemoveContainer" containerID="aa05e9c8dee4d9677811319ad135ee7caddba6ca16aedf281cdfb89e04f99e97"
Nov 03 00:40:19.070440 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.070400    4550 scope.go:122] "RemoveContainer" containerID="7cbda1ecd3f2441e5dcc38d5eddb0b296c39cabe5a421698e44cceecc8f38d0e"
Nov 03 00:40:19.080028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:19.080008    4550 scope.go:122] "RemoveContainer" containerID="50d4622d6bb4a35b2086fd61bceeece723bcf029fe157a847ab16e642458e291"
Nov 03 00:40:20.212219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:20.212164    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:20.895921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:20.895886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:21.358974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:21.358640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:21.837396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:21.837364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:22.034961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:22.034926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:22.035806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:22.035777    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"]
Nov 03 00:40:22.045117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:22.045088    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8511496-vxzfm"]
Nov 03 00:40:23.735226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:23.735194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:23.819325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:23.819285    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad94918f-9834-4b99-91a0-d06ce448bd14" path="/var/lib/kubelet/pods/ad94918f-9834-4b99-91a0-d06ce448bd14/volumes"
Nov 03 00:40:24.582816 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:24.582771    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:24.767063 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:24.767025    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:40:25.280666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:25.280616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:25.374826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:25.374792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:25.764332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:25.764303    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:26.935353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:26.935322    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:27.035150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:27.035114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:27.744224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:27.744195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:29.563102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:29.563012    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:30.686704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:30.686671    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:31.388344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.388290    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:31.827096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.827066    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"]
Nov 03 00:40:31.827862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.827841    4550 cpu_manager.go:414] "containerMap: removing container" podUID="20a710bd-7cfd-4afa-a338-641fb96ca261" containerName="revoke-keys"
Nov 03 00:40:31.827924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.827867    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a710bd-7cfd-4afa-a338-641fb96ca261" containerName="revoke-keys"
Nov 03 00:40:31.828033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.828021    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a710bd-7cfd-4afa-a338-641fb96ca261" containerName="revoke-keys"
Nov 03 00:40:31.833314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.833291    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:31.848548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.848525    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:40:31.849098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.849078    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"]
Nov 03 00:40:31.858350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.858312    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:40:31.935448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.935402    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:31.935601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.935566    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7g7m\" (UniqueName: \"kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:31.935644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:31.935624    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.037545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.037511    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.037751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.037603    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7g7m\" (UniqueName: \"kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.037751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.037633    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.038231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.038204    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.040303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.040282    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.057440 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.057398    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7g7m\" (UniqueName: \"kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m\") pod \"maas-api-revoke-sub-3089eaa7-f4f8c\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.149885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.149858    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:32.452925 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:40:32.452899    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29cf92c_3f62_43ba_88ba_f3fd4c553860.slice/crio-f530f7c24a052df94fb99be48afcbc615ff323901de58acf4f68215189664fdd WatchSource:0}: Error finding container f530f7c24a052df94fb99be48afcbc615ff323901de58acf4f68215189664fdd: Status 404 returned error can't find the container with id f530f7c24a052df94fb99be48afcbc615ff323901de58acf4f68215189664fdd
Nov 03 00:40:32.453693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:32.453631    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"]
Nov 03 00:40:33.134556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:33.134466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:33.161067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:33.161035    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c" event={"ID":"f29cf92c-3f62-43ba-88ba-f3fd4c553860","Type":"ContainerStarted","Data":"1efb07aa44d8eee7e3cf02c65cd2a640d458884ebc686252b4c484388c9f1b96"}
Nov 03 00:40:33.161228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:33.161103    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c" event={"ID":"f29cf92c-3f62-43ba-88ba-f3fd4c553860","Type":"ContainerStarted","Data":"f530f7c24a052df94fb99be48afcbc615ff323901de58acf4f68215189664fdd"}
Nov 03 00:40:33.222772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:33.222692    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c" podStartSLOduration=2.222669586 podStartE2EDuration="2.222669586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:33.20971818 +0000 UTC m=+2115.531255010" watchObservedRunningTime="2026-11-03 00:40:33.222669586 +0000 UTC m=+2115.544206385"
Nov 03 00:40:34.169361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:34.169327    4550 generic.go:356] "Generic (PLEG): container finished" podID="f29cf92c-3f62-43ba-88ba-f3fd4c553860" containerID="1efb07aa44d8eee7e3cf02c65cd2a640d458884ebc686252b4c484388c9f1b96" exitCode=0
Nov 03 00:40:34.169818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:34.169460    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c" event={"ID":"f29cf92c-3f62-43ba-88ba-f3fd4c553860","Type":"ContainerDied","Data":"1efb07aa44d8eee7e3cf02c65cd2a640d458884ebc686252b4c484388c9f1b96"}
Nov 03 00:40:34.911941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:34.911903    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:35.336724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.336693    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:35.498785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.498704    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca\") pod \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") "
Nov 03 00:40:35.498971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.498901    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m\" (UniqueName: \"kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m\") pod \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") "
Nov 03 00:40:35.499045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.499032    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca\") pod \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\" (UID: \"f29cf92c-3f62-43ba-88ba-f3fd4c553860\") "
Nov 03 00:40:35.499509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.499479    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca" pod "f29cf92c-3f62-43ba-88ba-f3fd4c553860" (UID: "f29cf92c-3f62-43ba-88ba-f3fd4c553860"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:35.499654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.499598    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:35.501112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.501080    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m" pod "f29cf92c-3f62-43ba-88ba-f3fd4c553860" (UID: "f29cf92c-3f62-43ba-88ba-f3fd4c553860"). InnerVolumeSpecName "kube-api-access-q7g7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:35.501112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.501104    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca" pod "f29cf92c-3f62-43ba-88ba-f3fd4c553860" (UID: "f29cf92c-3f62-43ba-88ba-f3fd4c553860"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:35.601197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.601163    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29cf92c-3f62-43ba-88ba-f3fd4c553860-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:35.601197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:35.601190    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7g7m\" (UniqueName: \"kubernetes.io/projected/f29cf92c-3f62-43ba-88ba-f3fd4c553860-kube-api-access-q7g7m\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:36.184628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:36.184598    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"
Nov 03 00:40:36.184817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:36.184635    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c" event={"ID":"f29cf92c-3f62-43ba-88ba-f3fd4c553860","Type":"ContainerDied","Data":"f530f7c24a052df94fb99be48afcbc615ff323901de58acf4f68215189664fdd"}
Nov 03 00:40:36.184817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:36.184693    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f530f7c24a052df94fb99be48afcbc615ff323901de58acf4f68215189664fdd"
Nov 03 00:40:36.530636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:36.530554    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:36.880803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:36.880726    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"]
Nov 03 00:40:36.881013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:36.880988    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy" containerID="cri-o://33af71c9369bc9440c3424ed3bd714b6ad16348301950e311b82a2f7feef6f3d" gracePeriod=30
Nov 03 00:40:38.230559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:38.230518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:39.897249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:39.897209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:41.531262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:41.531219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:41.650139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:41.650107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:41.975259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:41.975232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:42.141872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.141837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:42.249293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.249253    4550 generic.go:356] "Generic (PLEG): container finished" podID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerID="33af71c9369bc9440c3424ed3bd714b6ad16348301950e311b82a2f7feef6f3d" exitCode=0
Nov 03 00:40:42.249729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.249702    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" event={"ID":"76f6ba3d-68f1-4e5f-b97d-238cc5b98411","Type":"ContainerDied","Data":"33af71c9369bc9440c3424ed3bd714b6ad16348301950e311b82a2f7feef6f3d"}
Nov 03 00:40:42.280682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.280648    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"]
Nov 03 00:40:42.281740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.281713    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f29cf92c-3f62-43ba-88ba-f3fd4c553860" containerName="revoke-keys"
Nov 03 00:40:42.281893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.281878    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29cf92c-3f62-43ba-88ba-f3fd4c553860" containerName="revoke-keys"
Nov 03 00:40:42.282137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.282122    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29cf92c-3f62-43ba-88ba-f3fd4c553860" containerName="revoke-keys"
Nov 03 00:40:42.302452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.299131    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.317574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.317465    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"]
Nov 03 00:40:42.345749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.345701    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:40:42.354647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.354592    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:40:42.424924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.424887    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4j9\" (UniqueName: \"kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.425083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.424929    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.425204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.425181    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.427769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.427750    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:40:42.526679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.526591    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.527321 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.527299    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.527828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.527808    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.528206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.527183    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528373    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528188    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528556    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528586    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528619    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528653    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.528679    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert\") pod \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\" (UID: \"76f6ba3d-68f1-4e5f-b97d-238cc5b98411\") "
Nov 03 00:40:42.529336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.529161    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:42.529882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.529385    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.530355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.527682    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.530355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530147    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.530636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530410    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4j9\" (UniqueName: \"kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.530636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530461    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.530636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530607    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.530636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530624    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.530874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530638    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.530874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530652    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.530874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.530664    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.531366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.531337    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.533950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.533924    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.534060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.534033    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:42.534681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.534647    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "kube-api-access-87pf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.534793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.534754    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.535953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.535919    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy" pod "76f6ba3d-68f1-4e5f-b97d-238cc5b98411" (UID: "76f6ba3d-68f1-4e5f-b97d-238cc5b98411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.553304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.553275    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4j9\" (UniqueName: \"kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9\") pod \"maas-api-revoke-sub-57bd807b-t52pk\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.629579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.629538    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:42.632214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.632183    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.632214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.632212    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.632425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.632221    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87pf6\" (UniqueName: \"kubernetes.io/projected/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-kube-api-access-87pf6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.632425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.632230    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76f6ba3d-68f1-4e5f-b97d-238cc5b98411-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.965265 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:42.965238    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"]
Nov 03 00:40:43.257634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.257323    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk" event={"ID":"3e01db93-cbac-48a5-a6c3-02671b416605","Type":"ContainerStarted","Data":"c034d4a0ee183481606231cfcbaa5592dc40c03b0f5a511a92494c813423cf30"}
Nov 03 00:40:43.259215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.259191    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg" event={"ID":"76f6ba3d-68f1-4e5f-b97d-238cc5b98411","Type":"ContainerDied","Data":"4e6a5fad1cd3f480f5f2265540a3d2bf084a6df22072fdce5f45404e9913e997"}
Nov 03 00:40:43.259370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.259248    4550 scope.go:122] "RemoveContainer" containerID="33af71c9369bc9440c3424ed3bd714b6ad16348301950e311b82a2f7feef6f3d"
Nov 03 00:40:43.259370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.259308    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"
Nov 03 00:40:43.329990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.329954    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"]
Nov 03 00:40:43.340172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.340134    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qshkg"]
Nov 03 00:40:43.649778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.649755    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:43.808393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:43.808359    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" path="/var/lib/kubelet/pods/76f6ba3d-68f1-4e5f-b97d-238cc5b98411/volumes"
Nov 03 00:40:44.269614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:44.269521    4550 generic.go:356] "Generic (PLEG): container finished" podID="3e01db93-cbac-48a5-a6c3-02671b416605" containerID="033a37b02c905143bd867afb92d8ca254f056ef0c108b6e7e0f62b4e87bfc69a" exitCode=0
Nov 03 00:40:44.269800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:44.269602    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk" event={"ID":"3e01db93-cbac-48a5-a6c3-02671b416605","Type":"ContainerDied","Data":"033a37b02c905143bd867afb92d8ca254f056ef0c108b6e7e0f62b4e87bfc69a"}
Nov 03 00:40:45.435700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.435671    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:45.438239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.438209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:45.584236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.583055    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca\") pod \"3e01db93-cbac-48a5-a6c3-02671b416605\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") "
Nov 03 00:40:45.584236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.583193    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9\" (UniqueName: \"kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9\") pod \"3e01db93-cbac-48a5-a6c3-02671b416605\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") "
Nov 03 00:40:45.584236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.583226    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca\") pod \"3e01db93-cbac-48a5-a6c3-02671b416605\" (UID: \"3e01db93-cbac-48a5-a6c3-02671b416605\") "
Nov 03 00:40:45.589360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.584481    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca" pod "3e01db93-cbac-48a5-a6c3-02671b416605" (UID: "3e01db93-cbac-48a5-a6c3-02671b416605"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:45.589360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.586646    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:45.596935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.596900    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca" pod "3e01db93-cbac-48a5-a6c3-02671b416605" (UID: "3e01db93-cbac-48a5-a6c3-02671b416605"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:45.597084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.596940    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9" pod "3e01db93-cbac-48a5-a6c3-02671b416605" (UID: "3e01db93-cbac-48a5-a6c3-02671b416605"). InnerVolumeSpecName "kube-api-access-9h4j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:45.687928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.687890    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4j9\" (UniqueName: \"kubernetes.io/projected/3e01db93-cbac-48a5-a6c3-02671b416605-kube-api-access-9h4j9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:45.687928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.687922    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e01db93-cbac-48a5-a6c3-02671b416605-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:45.899792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:45.899755    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:46.023796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:46.023747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:46.286046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:46.285961    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"
Nov 03 00:40:46.286046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:46.285991    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk" event={"ID":"3e01db93-cbac-48a5-a6c3-02671b416605","Type":"ContainerDied","Data":"c034d4a0ee183481606231cfcbaa5592dc40c03b0f5a511a92494c813423cf30"}
Nov 03 00:40:46.286046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:46.286040    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c034d4a0ee183481606231cfcbaa5592dc40c03b0f5a511a92494c813423cf30"
Nov 03 00:40:47.405260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:47.405225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:47.594947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:47.594897    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:47.697085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:47.696979    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:49.118914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:49.118880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:50.748888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:50.748855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:50.832857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:50.832822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:50.932720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:50.932679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:51.978804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:51.978766    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"]
Nov 03 00:40:51.979295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:51.979012    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy" containerID="cri-o://93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059" gracePeriod=30
Nov 03 00:40:51.992689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:51.992656    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"]
Nov 03 00:40:51.993105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:51.993053    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy" containerID="cri-o://77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08" gracePeriod=30
Nov 03 00:40:51.996490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:51.996465    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"]
Nov 03 00:40:51.996726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:51.996659    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy" containerID="cri-o://6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b" gracePeriod=30
Nov 03 00:40:52.591332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:52.591295    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:54.321484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:54.321452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:54.511219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:54.511175    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:54.766024 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:54.765957    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:40:56.052847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:56.052813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:57.290878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.290850    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:40:57.349534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.349502    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:40:57.353729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.353590    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:40:57.368476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.368447    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.368745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.368724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.368933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.368896    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.369059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.368941    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.369122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.369060    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.369122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.369094    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.369200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.369140    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.369200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.369170    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.369307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.369291    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data\") pod \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\" (UID: \"5baea46a-4f16-487c-9606-b3eb9e9ed71e\") "
Nov 03 00:40:57.370813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.370604    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371363    4550 generic.go:356] "Generic (PLEG): container finished" podID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerID="6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b" exitCode=0
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371391    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371539    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371669    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" event={"ID":"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195","Type":"ContainerDied","Data":"6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b"}
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371709    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l" event={"ID":"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195","Type":"ContainerDied","Data":"c10fb4cbabfd31ddcad0bdb635af8512b7a5e29bb38cc0cbe0369bdc34a30d55"}
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371739    4550 scope.go:122] "RemoveContainer" containerID="6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b"
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.371942    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.372349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.372299    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.375680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.375641    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:57.376335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.376274    4550 generic.go:356] "Generic (PLEG): container finished" podID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerID="93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059" exitCode=0
Nov 03 00:40:57.376467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.376374    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"
Nov 03 00:40:57.376653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.376562    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" event={"ID":"5baea46a-4f16-487c-9606-b3eb9e9ed71e","Type":"ContainerDied","Data":"93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059"}
Nov 03 00:40:57.376653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.376611    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz" event={"ID":"5baea46a-4f16-487c-9606-b3eb9e9ed71e","Type":"ContainerDied","Data":"a3f9f944e0a418cfdf29fdd6912b96fa34fa8147f9d08a5768646dbac25d902a"}
Nov 03 00:40:57.380556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.378837    4550 generic.go:356] "Generic (PLEG): container finished" podID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerID="77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08" exitCode=0
Nov 03 00:40:57.380556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.380117    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" event={"ID":"de5fbc04-e9ea-47c0-ae97-b82a8bda8522","Type":"ContainerDied","Data":"77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08"}
Nov 03 00:40:57.380556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.380157    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2" event={"ID":"de5fbc04-e9ea-47c0-ae97-b82a8bda8522","Type":"ContainerDied","Data":"bbc691093d3eb655097018840e0dc49dafb4b06b9b835b83cfa4084904f3e99c"}
Nov 03 00:40:57.380556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.380227    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"
Nov 03 00:40:57.384760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.384692    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "kube-api-access-82rlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:57.384760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.384709    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:57.387369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.387334    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:57.387474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.387461    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy" pod "5baea46a-4f16-487c-9606-b3eb9e9ed71e" (UID: "5baea46a-4f16-487c-9606-b3eb9e9ed71e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.388363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.388280    4550 scope.go:122] "RemoveContainer" containerID="6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b"
Nov 03 00:40:57.388678 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:57.388636    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b\": container with ID starting with 6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b not found: ID does not exist" containerID="6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b"
Nov 03 00:40:57.388763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.388670    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b"} err="failed to get container status \"6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b\": rpc error: code = NotFound desc = could not find container \"6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b\": container with ID starting with 6eb07edd1d3923852f71d22bfc53d6868e218dd3e422d3e48a341fcdd979469b not found: ID does not exist"
Nov 03 00:40:57.388763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.388694    4550 scope.go:122] "RemoveContainer" containerID="93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059"
Nov 03 00:40:57.405274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.405243    4550 scope.go:122] "RemoveContainer" containerID="93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059"
Nov 03 00:40:57.405921 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:57.405887    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059\": container with ID starting with 93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059 not found: ID does not exist" containerID="93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059"
Nov 03 00:40:57.406047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.405931    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059"} err="failed to get container status \"93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059\": rpc error: code = NotFound desc = could not find container \"93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059\": container with ID starting with 93fb718beddbb950f0c1918327c06e3cac2cc99aea59d9b3bead9cd76168c059 not found: ID does not exist"
Nov 03 00:40:57.406047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.405957    4550 scope.go:122] "RemoveContainer" containerID="77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08"
Nov 03 00:40:57.437367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.437345    4550 scope.go:122] "RemoveContainer" containerID="77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08"
Nov 03 00:40:57.437752 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:40:57.437729    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08\": container with ID starting with 77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08 not found: ID does not exist" containerID="77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08"
Nov 03 00:40:57.437818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.437765    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08"} err="failed to get container status \"77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08\": rpc error: code = NotFound desc = could not find container \"77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08\": container with ID starting with 77a9c8e1b0da17ccee0c867eb2b866c031166329fc8e4a05c0c6f15ed0f79c08 not found: ID does not exist"
Nov 03 00:40:57.471484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471448    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.471676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471522    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.471676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471552    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.471676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471587    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.471676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471613    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.471910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471700    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.471910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471751    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.471910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471787    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.471910 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471862    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.472114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471911    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.472114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471912    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.472114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.471965    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.472114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472013    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.472114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472065    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.472114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472113    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.472521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472153    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.472521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472192    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket\") pod \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\" (UID: \"de5fbc04-e9ea-47c0-ae97-b82a8bda8522\") "
Nov 03 00:40:57.472521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472226    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.472521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472274    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy\") pod \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\" (UID: \"9c2d1d73-bb05-432f-bb46-e9e0f8c2f195\") "
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472962    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.472988    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473002    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473016    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rlk\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-kube-api-access-82rlk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473030    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473043    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473055    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473067    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473079    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.473396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.473093    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5baea46a-4f16-487c-9606-b3eb9e9ed71e-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.474181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.474153    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.474292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.474238    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.475790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.474497    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.475790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.474524    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:57.475790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.474761    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.475790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.474828    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:57.475790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.475078    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.477862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.477557    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.477862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.477639    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:57.477862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.477679    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "kube-api-access-bk8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:57.477862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.477830    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.478338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.478309    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:57.479096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.479077    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "kube-api-access-hhdnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:57.479601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.479530    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.481023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.480991    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:57.482232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.482207    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token" pod "de5fbc04-e9ea-47c0-ae97-b82a8bda8522" (UID: "de5fbc04-e9ea-47c0-ae97-b82a8bda8522"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:57.482419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.482396    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo" pod "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" (UID: "9c2d1d73-bb05-432f-bb46-e9e0f8c2f195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574593    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574624    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk8sj\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-kube-api-access-bk8sj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574634    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574643    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhdnm\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-kube-api-access-hhdnm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574654    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574664    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574671    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.574675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574681    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574689    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574697    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574704    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574711    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574718    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574725    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574733    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574740    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.575087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.574747    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5fbc04-e9ea-47c0-ae97-b82a8bda8522-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:40:57.720591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.720551    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"]
Nov 03 00:40:57.733658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.733623    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-k5c9l"]
Nov 03 00:40:57.768824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.768786    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"]
Nov 03 00:40:57.774823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.774783    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-qqtxz"]
Nov 03 00:40:57.810975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.810937    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" path="/var/lib/kubelet/pods/5baea46a-4f16-487c-9606-b3eb9e9ed71e/volumes"
Nov 03 00:40:57.813478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.811744    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" path="/var/lib/kubelet/pods/9c2d1d73-bb05-432f-bb46-e9e0f8c2f195/volumes"
Nov 03 00:40:57.813478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.812165    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"]
Nov 03 00:40:57.817215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.817192    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-6qch2"]
Nov 03 00:40:57.848061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:57.847967    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:59.718770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:59.718734    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:40:59.809914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:40:59.809882    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" path="/var/lib/kubelet/pods/de5fbc04-e9ea-47c0-ae97-b82a8bda8522/volumes"
Nov 03 00:41:01.437822 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:01.437787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:03.224087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:03.224048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:04.995729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:04.995676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:06.609845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:06.609809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:06.977356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:06.977325    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"]
Nov 03 00:41:06.977619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:06.977595    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy" containerID="cri-o://bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57" gracePeriod=30
Nov 03 00:41:07.067637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:07.067601    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"]
Nov 03 00:41:07.067866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:07.067841    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy" containerID="cri-o://7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38" gracePeriod=30
Nov 03 00:41:08.326033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:08.326000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:09.935878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:09.935844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:11.547493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:11.547464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:11.628829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:11.628791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:11.733106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:11.733075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:12.293619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.293591    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381404    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381510    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381566    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381624    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381662    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381690    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381700    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381758    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381847    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.381880    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo\") pod \"b5b65012-d45b-4d41-adf9-c9996ebe77db\" (UID: \"b5b65012-d45b-4d41-adf9-c9996ebe77db\") "
Nov 03 00:41:12.382866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.382575    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.384169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.384142    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "kube-api-access-tgntw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.384562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.384513    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.384761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.384666    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.384761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.384725    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.384761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.384739    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.384924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.384831    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.385636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.385614    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.387313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.387289    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy" pod "b5b65012-d45b-4d41-adf9-c9996ebe77db" (UID: "b5b65012-d45b-4d41-adf9-c9996ebe77db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.412286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.407659    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.485917    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.485989    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486029    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486088    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486127    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486152    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486265    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486351    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.486479    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token\") pod \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\" (UID: \"b5fc7b6f-1906-4411-afbc-6efaf10272b8\") "
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487031    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b65012-d45b-4d41-adf9-c9996ebe77db-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487053    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487067    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487080    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487092    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487105    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b65012-d45b-4d41-adf9-c9996ebe77db-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487116    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b65012-d45b-4d41-adf9-c9996ebe77db-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.487562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.487130    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgntw\" (UniqueName: \"kubernetes.io/projected/b5b65012-d45b-4d41-adf9-c9996ebe77db-kube-api-access-tgntw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.490499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.489896    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.490499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.489912    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.490499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.490139    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.490499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.490220    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.490499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.490472    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.490864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.490610    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.492409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.492254    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.492409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.492353    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "kube-api-access-h8vzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.492687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.492652    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo" pod "b5fc7b6f-1906-4411-afbc-6efaf10272b8" (UID: "b5fc7b6f-1906-4411-afbc-6efaf10272b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.493596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.493573    4550 generic.go:356] "Generic (PLEG): container finished" podID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerID="bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57" exitCode=0
Nov 03 00:41:12.493689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.493642    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" event={"ID":"b5b65012-d45b-4d41-adf9-c9996ebe77db","Type":"ContainerDied","Data":"bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57"}
Nov 03 00:41:12.493689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.493681    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz" event={"ID":"b5b65012-d45b-4d41-adf9-c9996ebe77db","Type":"ContainerDied","Data":"3fe1aa4bb0fc5b52bfb565969e05d4520e32727650223408ba0a2dbb78d345cc"}
Nov 03 00:41:12.493793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.493700    4550 scope.go:122] "RemoveContainer" containerID="bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57"
Nov 03 00:41:12.493848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.493807    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"
Nov 03 00:41:12.496541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.496514    4550 generic.go:356] "Generic (PLEG): container finished" podID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerID="7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38" exitCode=0
Nov 03 00:41:12.496639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.496611    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" event={"ID":"b5fc7b6f-1906-4411-afbc-6efaf10272b8","Type":"ContainerDied","Data":"7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38"}
Nov 03 00:41:12.496702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.496653    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2" event={"ID":"b5fc7b6f-1906-4411-afbc-6efaf10272b8","Type":"ContainerDied","Data":"239b2efc8ca94ff642347129748dc10cd7c38bda5b314eb2145c5a3809c52a88"}
Nov 03 00:41:12.496762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.496720    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"
Nov 03 00:41:12.516116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.516098    4550 scope.go:122] "RemoveContainer" containerID="bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57"
Nov 03 00:41:12.517086 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:41:12.516409    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57\": container with ID starting with bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57 not found: ID does not exist" containerID="bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57"
Nov 03 00:41:12.517086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.516465    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57"} err="failed to get container status \"bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57\": rpc error: code = NotFound desc = could not find container \"bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57\": container with ID starting with bb0482fa5ad4d2d30a7e5040ac42cdee5ae4c70e982b5ed5eec89388dc287e57 not found: ID does not exist"
Nov 03 00:41:12.517086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.516491    4550 scope.go:122] "RemoveContainer" containerID="7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38"
Nov 03 00:41:12.526617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.526534    4550 scope.go:122] "RemoveContainer" containerID="7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38"
Nov 03 00:41:12.526972 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:41:12.526900    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38\": container with ID starting with 7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38 not found: ID does not exist" containerID="7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38"
Nov 03 00:41:12.526972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.526933    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38"} err="failed to get container status \"7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38\": rpc error: code = NotFound desc = could not find container \"7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38\": container with ID starting with 7f12c13fa41dd34d234a42e87c1e66beb0727b674a75d342221d96de40b44c38 not found: ID does not exist"
Nov 03 00:41:12.554899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.554845    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"]
Nov 03 00:41:12.576481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.561557    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-8jrtz"]
Nov 03 00:41:12.588829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588800    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588846    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8vzg\" (UniqueName: \"kubernetes.io/projected/b5fc7b6f-1906-4411-afbc-6efaf10272b8-kube-api-access-h8vzg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588861    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588875    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588889    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588914    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588928    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588940    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5fc7b6f-1906-4411-afbc-6efaf10272b8-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.588974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.588953    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5fc7b6f-1906-4411-afbc-6efaf10272b8-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.609228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.609191    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"]
Nov 03 00:41:12.613673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:12.613646    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-km9q2"]
Nov 03 00:41:13.381010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:13.380939    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:13.820655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:13.820618    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" path="/var/lib/kubelet/pods/b5b65012-d45b-4d41-adf9-c9996ebe77db/volumes"
Nov 03 00:41:13.821231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:13.821211    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" path="/var/lib/kubelet/pods/b5fc7b6f-1906-4411-afbc-6efaf10272b8/volumes"
Nov 03 00:41:15.109140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:15.109104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:15.227853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:15.227819    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:16.940627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:16.940592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:18.576645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:18.576593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:18.682963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:18.682930    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:19.108224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.108188    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:19.146143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.146120    4550 scope.go:122] "RemoveContainer" containerID="ab21ecceff5a480bed98b3391fd408d74ef70c825ae3bed3c99c602d208b731c"
Nov 03 00:41:19.156561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.156541    4550 scope.go:122] "RemoveContainer" containerID="d9a72c10b180f38def910aba1d456eeaeb4956160be135d609830e88e6b12b97"
Nov 03 00:41:19.167928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.167905    4550 scope.go:122] "RemoveContainer" containerID="5eda23a45ef28e00fbadb76aabf00b52d41b4033a884db7c6792bd6c1ead8654"
Nov 03 00:41:19.176808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.176779    4550 scope.go:122] "RemoveContainer" containerID="ed324104f3d4e4b824937cc679da113a4ac1a41b65b808acc6038fb74bd50999"
Nov 03 00:41:19.187210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.186866    4550 scope.go:122] "RemoveContainer" containerID="7475aef6335feb0296dd20a4c9e1ec4e55e8cd2cb28223de6628128e6cf4a89d"
Nov 03 00:41:19.216238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.216164    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:19.282658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:19.282636    4550 scope.go:122] "RemoveContainer" containerID="f7b3d40bbef545b16fcaaac3ef199f90fc1f47118ead92e9346b8fbc0717c14b"
Nov 03 00:41:20.327683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:20.327653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:21.301496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:21.301460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:21.750923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:21.750892    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:22.263827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:22.263797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:22.551494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:22.550543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:22.683424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:22.683395    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:22.811844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:22.811768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:22.924285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:22.924249    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:24.293911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:24.293872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:24.388751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:24.388718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:24.494361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:24.494329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:24.724997 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:41:24.724956    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:41:25.731778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:25.731741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:25.834010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:25.833972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:26.874715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:26.874683    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:27.514080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:27.514048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:27.603285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:27.603256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:28.932248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:28.932210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:30.721734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.721705    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:30.883695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.883658    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:30.931547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931517    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"]
Nov 03 00:41:30.931957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931944    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931959    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931970    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931975    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931981    4550 cpu_manager.go:414] "containerMap: removing container" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.931985    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932007    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3e01db93-cbac-48a5-a6c3-02671b416605" containerName="revoke-keys"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932011    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e01db93-cbac-48a5-a6c3-02671b416605" containerName="revoke-keys"
Nov 03 00:41:30.932017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932018    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932022    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932032    4550 cpu_manager.go:414] "containerMap: removing container" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932037    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932045    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932049    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932134    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5fc7b6f-1906-4411-afbc-6efaf10272b8" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932143    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="76f6ba3d-68f1-4e5f-b97d-238cc5b98411" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932152    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e01db93-cbac-48a5-a6c3-02671b416605" containerName="revoke-keys"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932157    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5baea46a-4f16-487c-9606-b3eb9e9ed71e" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932164    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5fbc04-e9ea-47c0-ae97-b82a8bda8522" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932168    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b65012-d45b-4d41-adf9-c9996ebe77db" containerName="istio-proxy"
Nov 03 00:41:30.932326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.932175    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2d1d73-bb05-432f-bb46-e9e0f8c2f195" containerName="istio-proxy"
Nov 03 00:41:30.937597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.937580    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:30.954767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.954744    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"]
Nov 03 00:41:30.964574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.964547    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:41:30.964708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.964616    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:41:30.975764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:30.975686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:31.010098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.010077    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.010234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.010134    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvg8p\" (UniqueName: \"kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.010298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.010230    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.111257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.111225    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.111406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.111271    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvg8p\" (UniqueName: \"kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.111406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.111295    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.111914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.111890    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.113529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.113509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.130908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.130880    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvg8p\" (UniqueName: \"kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p\") pod \"maas-api-revoke-sub-ab35fe9d-8mbpb\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.251128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.251062    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:31.453760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.453731    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"]
Nov 03 00:41:31.458629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.458609    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.476860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.476832    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"]
Nov 03 00:41:31.516047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.515978    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.516177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.516153    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.516242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.516186    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqcc\" (UniqueName: \"kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.559510 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:41:31.559483    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04155849_815d_47af_81d2_e8bc8683426d.slice/crio-4f57eb206d014b99eeadb6db0bad012a69785325dc71daf7e6cd4db5b50b932b WatchSource:0}: Error finding container 4f57eb206d014b99eeadb6db0bad012a69785325dc71daf7e6cd4db5b50b932b: Status 404 returned error can't find the container with id 4f57eb206d014b99eeadb6db0bad012a69785325dc71daf7e6cd4db5b50b932b
Nov 03 00:41:31.560944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.560923    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"]
Nov 03 00:41:31.617819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.617798    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.617933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.617882    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.617933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.617900    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqcc\" (UniqueName: \"kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.618597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.618578    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.620089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.620068    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.638135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.638114    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqcc\" (UniqueName: \"kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc\") pod \"maas-api-revoke-sub-e51c09a6-96vdk\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:31.699247 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.699214    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb" event={"ID":"04155849-815d-47af-81d2-e8bc8683426d","Type":"ContainerStarted","Data":"4f57eb206d014b99eeadb6db0bad012a69785325dc71daf7e6cd4db5b50b932b"}
Nov 03 00:41:31.772298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:31.772274    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:32.070180 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:41:32.070152    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d95e483_63cd_4e6c_a0f1_085254a10250.slice/crio-68da58561f2d31a95a79d75c4f2abdcc32a7167b042d1224008ff0a165063825 WatchSource:0}: Error finding container 68da58561f2d31a95a79d75c4f2abdcc32a7167b042d1224008ff0a165063825: Status 404 returned error can't find the container with id 68da58561f2d31a95a79d75c4f2abdcc32a7167b042d1224008ff0a165063825
Nov 03 00:41:32.070841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.070776    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"]
Nov 03 00:41:32.409176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.409150    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:32.708258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.708217    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk" event={"ID":"6d95e483-63cd-4e6c-a0f1-085254a10250","Type":"ContainerStarted","Data":"a1c61fcd37e1faa5339d5b69444a037ca6bdef3f82e0666bb5f6d845ad5f6c08"}
Nov 03 00:41:32.708486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.708286    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk" event={"ID":"6d95e483-63cd-4e6c-a0f1-085254a10250","Type":"ContainerStarted","Data":"68da58561f2d31a95a79d75c4f2abdcc32a7167b042d1224008ff0a165063825"}
Nov 03 00:41:32.709809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.709786    4550 generic.go:356] "Generic (PLEG): container finished" podID="04155849-815d-47af-81d2-e8bc8683426d" containerID="2c3efa1f3e9114582e9523c81bf872678a688c6474b679b52bc50c9d4ef225d6" exitCode=0
Nov 03 00:41:32.709912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.709815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb" event={"ID":"04155849-815d-47af-81d2-e8bc8683426d","Type":"ContainerDied","Data":"2c3efa1f3e9114582e9523c81bf872678a688c6474b679b52bc50c9d4ef225d6"}
Nov 03 00:41:32.748540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:32.748495    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk" podStartSLOduration=1.7484843479999999 podStartE2EDuration="1.748484348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:32.746990724 +0000 UTC m=+2175.068527522" watchObservedRunningTime="2026-11-03 00:41:32.748484348 +0000 UTC m=+2175.070021147"
Nov 03 00:41:33.720493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:33.720447    4550 generic.go:356] "Generic (PLEG): container finished" podID="6d95e483-63cd-4e6c-a0f1-085254a10250" containerID="a1c61fcd37e1faa5339d5b69444a037ca6bdef3f82e0666bb5f6d845ad5f6c08" exitCode=0
Nov 03 00:41:33.724720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:33.720588    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk" event={"ID":"6d95e483-63cd-4e6c-a0f1-085254a10250","Type":"ContainerDied","Data":"a1c61fcd37e1faa5339d5b69444a037ca6bdef3f82e0666bb5f6d845ad5f6c08"}
Nov 03 00:41:33.895014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:33.894987    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:34.045345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.045267    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca\") pod \"04155849-815d-47af-81d2-e8bc8683426d\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") "
Nov 03 00:41:34.045546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.045379    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p\" (UniqueName: \"kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p\") pod \"04155849-815d-47af-81d2-e8bc8683426d\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") "
Nov 03 00:41:34.045546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.045439    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca\") pod \"04155849-815d-47af-81d2-e8bc8683426d\" (UID: \"04155849-815d-47af-81d2-e8bc8683426d\") "
Nov 03 00:41:34.045693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.045665    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca" pod "04155849-815d-47af-81d2-e8bc8683426d" (UID: "04155849-815d-47af-81d2-e8bc8683426d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:34.045826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.045808    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04155849-815d-47af-81d2-e8bc8683426d-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:34.047409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.047386    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p" pod "04155849-815d-47af-81d2-e8bc8683426d" (UID: "04155849-815d-47af-81d2-e8bc8683426d"). InnerVolumeSpecName "kube-api-access-rvg8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:34.047409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.047390    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca" pod "04155849-815d-47af-81d2-e8bc8683426d" (UID: "04155849-815d-47af-81d2-e8bc8683426d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:34.117090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.117055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:34.147222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.147201    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04155849-815d-47af-81d2-e8bc8683426d-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:34.147222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.147220    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvg8p\" (UniqueName: \"kubernetes.io/projected/04155849-815d-47af-81d2-e8bc8683426d-kube-api-access-rvg8p\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:34.731858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.731833    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"
Nov 03 00:41:34.732227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.731877    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb" event={"ID":"04155849-815d-47af-81d2-e8bc8683426d","Type":"ContainerDied","Data":"4f57eb206d014b99eeadb6db0bad012a69785325dc71daf7e6cd4db5b50b932b"}
Nov 03 00:41:34.732227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.731931    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f57eb206d014b99eeadb6db0bad012a69785325dc71daf7e6cd4db5b50b932b"
Nov 03 00:41:34.803057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.803027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:34.902268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:34.902242    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:35.057955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.057879    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca\") pod \"6d95e483-63cd-4e6c-a0f1-085254a10250\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") "
Nov 03 00:41:35.057955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.057939    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc\" (UniqueName: \"kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc\") pod \"6d95e483-63cd-4e6c-a0f1-085254a10250\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") "
Nov 03 00:41:35.058194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.058134    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca\") pod \"6d95e483-63cd-4e6c-a0f1-085254a10250\" (UID: \"6d95e483-63cd-4e6c-a0f1-085254a10250\") "
Nov 03 00:41:35.058553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.058502    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca" pod "6d95e483-63cd-4e6c-a0f1-085254a10250" (UID: "6d95e483-63cd-4e6c-a0f1-085254a10250"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:35.058668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.058593    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:35.060024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.059996    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca" pod "6d95e483-63cd-4e6c-a0f1-085254a10250" (UID: "6d95e483-63cd-4e6c-a0f1-085254a10250"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:35.060129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.060048    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc" pod "6d95e483-63cd-4e6c-a0f1-085254a10250" (UID: "6d95e483-63cd-4e6c-a0f1-085254a10250"). InnerVolumeSpecName "kube-api-access-vsqcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:35.159592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.159558    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d95e483-63cd-4e6c-a0f1-085254a10250-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:35.159592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.159583    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqcc\" (UniqueName: \"kubernetes.io/projected/6d95e483-63cd-4e6c-a0f1-085254a10250-kube-api-access-vsqcc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:35.733153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.733059    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:35.741175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.741151    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"
Nov 03 00:41:35.741175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.741159    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk" event={"ID":"6d95e483-63cd-4e6c-a0f1-085254a10250","Type":"ContainerDied","Data":"68da58561f2d31a95a79d75c4f2abdcc32a7167b042d1224008ff0a165063825"}
Nov 03 00:41:35.741349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:35.741199    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68da58561f2d31a95a79d75c4f2abdcc32a7167b042d1224008ff0a165063825"
Nov 03 00:41:36.568956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.568921    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"]
Nov 03 00:41:36.569708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.569688    4550 cpu_manager.go:414] "containerMap: removing container" podUID="04155849-815d-47af-81d2-e8bc8683426d" containerName="revoke-keys"
Nov 03 00:41:36.569780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.569711    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04155849-815d-47af-81d2-e8bc8683426d" containerName="revoke-keys"
Nov 03 00:41:36.569780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.569726    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6d95e483-63cd-4e6c-a0f1-085254a10250" containerName="revoke-keys"
Nov 03 00:41:36.569780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.569734    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d95e483-63cd-4e6c-a0f1-085254a10250" containerName="revoke-keys"
Nov 03 00:41:36.569927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.569857    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="04155849-815d-47af-81d2-e8bc8683426d" containerName="revoke-keys"
Nov 03 00:41:36.569927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.569869    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d95e483-63cd-4e6c-a0f1-085254a10250" containerName="revoke-keys"
Nov 03 00:41:36.573666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.573642    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.606830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.606806    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-099583-openshift-default-dockercfg-hbtbr\""
Nov 03 00:41:36.610615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.610587    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"]
Nov 03 00:41:36.677421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677381    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677661    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677693    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677791    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677889    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677927    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd56h\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.677982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.677973    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.678015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.678003    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.778988    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779033    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd56h\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779054    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779196    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779266    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779363    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779465    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779477    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779520    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779569    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779575    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.779877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779743    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.780109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.779980    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.780310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.780290    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.781593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.781572    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.782012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.781991    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.799025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.798993    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.799549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.799530    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd56h\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:36.891266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:36.891212    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:37.202203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.202168    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"]
Nov 03 00:41:37.202458 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:41:37.202419    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00091f86_a803_44e7_9aed_783ad4016fec.slice/crio-e6f00adc31132fc5cb9a6b0c69dd2e7f010cfd9b4114d94eb1180018a942dea3 WatchSource:0}: Error finding container e6f00adc31132fc5cb9a6b0c69dd2e7f010cfd9b4114d94eb1180018a942dea3: Status 404 returned error can't find the container with id e6f00adc31132fc5cb9a6b0c69dd2e7f010cfd9b4114d94eb1180018a942dea3
Nov 03 00:41:37.205358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.205315    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:37.205477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.205405    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:37.205524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.205494    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:37.308499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.308466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:37.446336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.446288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:37.754837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.754807    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" event={"ID":"00091f86-a803-44e7-9aed-783ad4016fec","Type":"ContainerStarted","Data":"1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105"}
Nov 03 00:41:37.755015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.754854    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" event={"ID":"00091f86-a803-44e7-9aed-783ad4016fec","Type":"ContainerStarted","Data":"e6f00adc31132fc5cb9a6b0c69dd2e7f010cfd9b4114d94eb1180018a942dea3"}
Nov 03 00:41:37.799054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.799002    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podStartSLOduration=1.79898593 podStartE2EDuration="1.79898593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:37.796948735 +0000 UTC m=+2180.118485533" watchObservedRunningTime="2026-11-03 00:41:37.79898593 +0000 UTC m=+2180.120522730"
Nov 03 00:41:37.894409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:37.894381    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:38.898766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:38.897132    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 00:41:38.898766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:38.897192    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 00:41:39.138629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:39.138592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:39.892376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:39.892334    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 00:41:39.892633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:39.892397    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 00:41:40.892093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:40.892060    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 00:41:40.892514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:40.892114    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 00:41:40.973455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:40.970469    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:41.313422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.313385    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"]
Nov 03 00:41:41.318346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.318322    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.334719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.334697    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"]
Nov 03 00:41:41.345691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.344583    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:41:41.345691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.344700    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:41:41.376453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.375680    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2h8v\" (UniqueName: \"kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.376453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.375772    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.376453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.375822    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.482228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.482193    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.482459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.482262    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.482541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.482508    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2h8v\" (UniqueName: \"kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.483185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.483161    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.485166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.485141    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.502235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.502214    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2h8v\" (UniqueName: \"kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v\") pod \"maas-api-revoke-sub-6a5ca5d9-twmjx\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.632392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.632324    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:41.883305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.883212    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"]
Nov 03 00:41:41.888905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.888884    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:41.892399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.892371    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 00:41:41.892794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.892466    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 00:41:41.901240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.900908    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"]
Nov 03 00:41:41.985808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.985776    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"]
Nov 03 00:41:41.997471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.997417    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:41.997754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.997735    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbds\" (UniqueName: \"kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:41.997938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:41.997913    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.099774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.099734    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.099955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.099846    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbds\" (UniqueName: \"kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.099955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.099871    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.100680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.100648    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.102423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.102402    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.120437 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.120406    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbds\" (UniqueName: \"kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds\") pod \"maas-api-revoke-sub-d94c0b7d-r9psc\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.209983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.209945    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:42.612123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.611950    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"]
Nov 03 00:41:42.806013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.805979    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc" event={"ID":"bfc36ba9-924b-4113-9ca7-4a8ed2db6458","Type":"ContainerStarted","Data":"70f76356e488c2d49f8bb307126709561e6bb555641c8fa95d6a568ad4688df1"}
Nov 03 00:41:42.807759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.807734    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx" event={"ID":"774409d7-ec27-43c9-812c-4687a4632105","Type":"ContainerStarted","Data":"6567494d5c238f65d6d0693f019aea036cfdf764a39ea23d58ff89edaec941ab"}
Nov 03 00:41:42.807863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.807773    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx" event={"ID":"774409d7-ec27-43c9-812c-4687a4632105","Type":"ContainerStarted","Data":"5b0803a4012c0d84d40ff71ab66faefd73c1ad0eb32af9ef439a8b714e3a203f"}
Nov 03 00:41:42.826669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.826638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:42.849367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.849271    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx" podStartSLOduration=1.849258627 podStartE2EDuration="1.849258627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:42.847456281 +0000 UTC m=+2185.168993078" watchObservedRunningTime="2026-11-03 00:41:42.849258627 +0000 UTC m=+2185.170795451"
Nov 03 00:41:42.913258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.913219    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body=
Nov 03 00:41:42.913773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:42.913279    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused"
Nov 03 00:41:43.815672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.815640    4550 generic.go:356] "Generic (PLEG): container finished" podID="bfc36ba9-924b-4113-9ca7-4a8ed2db6458" containerID="2a350f91660ddb95175a8acb50911d088a2573d63f6198c73ea4a0f9f1ff8041" exitCode=0
Nov 03 00:41:43.815848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.815738    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc" event={"ID":"bfc36ba9-924b-4113-9ca7-4a8ed2db6458","Type":"ContainerDied","Data":"2a350f91660ddb95175a8acb50911d088a2573d63f6198c73ea4a0f9f1ff8041"}
Nov 03 00:41:43.817413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.817389    4550 generic.go:356] "Generic (PLEG): container finished" podID="774409d7-ec27-43c9-812c-4687a4632105" containerID="6567494d5c238f65d6d0693f019aea036cfdf764a39ea23d58ff89edaec941ab" exitCode=0
Nov 03 00:41:43.817569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.817550    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx" event={"ID":"774409d7-ec27-43c9-812c-4687a4632105","Type":"ContainerDied","Data":"6567494d5c238f65d6d0693f019aea036cfdf764a39ea23d58ff89edaec941ab"}
Nov 03 00:41:43.898013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.897967    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:43.898013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.898694    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:43.900006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:43.899548    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:41:44.632105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:44.632073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:45.006969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.006938    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:45.041254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.041229    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds\" (UniqueName: \"kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds\") pod \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") "
Nov 03 00:41:45.041387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.041288    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca\") pod \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") "
Nov 03 00:41:45.041387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.041346    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca\") pod \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\" (UID: \"bfc36ba9-924b-4113-9ca7-4a8ed2db6458\") "
Nov 03 00:41:45.041740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.041705    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca" pod "bfc36ba9-924b-4113-9ca7-4a8ed2db6458" (UID: "bfc36ba9-924b-4113-9ca7-4a8ed2db6458"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:45.041917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.041894    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:45.043443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.043412    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds" pod "bfc36ba9-924b-4113-9ca7-4a8ed2db6458" (UID: "bfc36ba9-924b-4113-9ca7-4a8ed2db6458"). InnerVolumeSpecName "kube-api-access-slbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:45.043691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.043669    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca" pod "bfc36ba9-924b-4113-9ca7-4a8ed2db6458" (UID: "bfc36ba9-924b-4113-9ca7-4a8ed2db6458"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:45.067157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.067130    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:45.142942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.142864    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca\") pod \"774409d7-ec27-43c9-812c-4687a4632105\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") "
Nov 03 00:41:45.143106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.143053    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca\") pod \"774409d7-ec27-43c9-812c-4687a4632105\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") "
Nov 03 00:41:45.143171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.143139    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v\" (UniqueName: \"kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v\") pod \"774409d7-ec27-43c9-812c-4687a4632105\" (UID: \"774409d7-ec27-43c9-812c-4687a4632105\") "
Nov 03 00:41:45.143263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.143233    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca" pod "774409d7-ec27-43c9-812c-4687a4632105" (UID: "774409d7-ec27-43c9-812c-4687a4632105"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:45.143624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.143594    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbds\" (UniqueName: \"kubernetes.io/projected/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-kube-api-access-slbds\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:45.143624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.143615    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774409d7-ec27-43c9-812c-4687a4632105-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:45.143810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.143628    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfc36ba9-924b-4113-9ca7-4a8ed2db6458-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:45.145219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.145193    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca" pod "774409d7-ec27-43c9-812c-4687a4632105" (UID: "774409d7-ec27-43c9-812c-4687a4632105"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:45.145722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.145702    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v" pod "774409d7-ec27-43c9-812c-4687a4632105" (UID: "774409d7-ec27-43c9-812c-4687a4632105"). InnerVolumeSpecName "kube-api-access-b2h8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:45.244710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.244676    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774409d7-ec27-43c9-812c-4687a4632105-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:45.244710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.244706    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2h8v\" (UniqueName: \"kubernetes.io/projected/774409d7-ec27-43c9-812c-4687a4632105-kube-api-access-b2h8v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:45.831138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.831110    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"
Nov 03 00:41:45.831138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.831114    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc" event={"ID":"bfc36ba9-924b-4113-9ca7-4a8ed2db6458","Type":"ContainerDied","Data":"70f76356e488c2d49f8bb307126709561e6bb555641c8fa95d6a568ad4688df1"}
Nov 03 00:41:45.831619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.831167    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70f76356e488c2d49f8bb307126709561e6bb555641c8fa95d6a568ad4688df1"
Nov 03 00:41:45.832604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.832577    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx" event={"ID":"774409d7-ec27-43c9-812c-4687a4632105","Type":"ContainerDied","Data":"5b0803a4012c0d84d40ff71ab66faefd73c1ad0eb32af9ef439a8b714e3a203f"}
Nov 03 00:41:45.832710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.832603    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"
Nov 03 00:41:45.832710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:45.832623    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b0803a4012c0d84d40ff71ab66faefd73c1ad0eb32af9ef439a8b714e3a203f"
Nov 03 00:41:46.394959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:46.394920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:46.983595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:46.983560    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:48.008785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:48.008749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:48.258426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:48.258383    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:48.848485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:48.831913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:49.097419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:49.097371    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:49.771381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:49.771344    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:51.500014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:51.499975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:51.609043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:51.609009    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:51.715456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:51.712079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:52.298060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.298023    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"]
Nov 03 00:41:52.298270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.298221    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" podUID="f177d44a-571f-415d-9328-4936506943f5" containerName="discovery" containerID="cri-o://2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac" gracePeriod=30
Nov 03 00:41:52.309873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.309845    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"]
Nov 03 00:41:52.310101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.310064    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy" containerID="cri-o://64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66" gracePeriod=30
Nov 03 00:41:52.628313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.628284    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:41:52.652945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.652915    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653001    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653061    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653098    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653148    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653225    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653287    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap\") pod \"f177d44a-571f-415d-9328-4936506943f5\" (UID: \"f177d44a-571f-415d-9328-4936506943f5\") "
Nov 03 00:41:52.653904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.653853    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:52.654169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.654144    4550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f177d44a-571f-415d-9328-4936506943f5-istio-csr-ca-configmap\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.657642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.656161    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:52.657775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.657693    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:52.658159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.658041    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:52.658285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.658251    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:52.663504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.658744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:41:52.663504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.658855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8" pod "f177d44a-571f-415d-9328-4936506943f5" (UID: "f177d44a-571f-415d-9328-4936506943f5"). InnerVolumeSpecName "kube-api-access-ktcp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:52.755112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.755081    4550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-cacerts\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.755112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.755111    4550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-csr-dns-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.755112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.755121    4550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f177d44a-571f-415d-9328-4936506943f5-local-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.755356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.755132    4550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f177d44a-571f-415d-9328-4936506943f5-istio-kubeconfig\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.755356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.755145    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktcp8\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-kube-api-access-ktcp8\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.755356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.755155    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f177d44a-571f-415d-9328-4936506943f5-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:52.825278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.825248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:52.931191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.931157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:52.970657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.970621    4550 generic.go:356] "Generic (PLEG): container finished" podID="f177d44a-571f-415d-9328-4936506943f5" containerID="2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac" exitCode=0
Nov 03 00:41:52.970850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.970668    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"
Nov 03 00:41:52.970850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.970691    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" event={"ID":"f177d44a-571f-415d-9328-4936506943f5","Type":"ContainerDied","Data":"2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac"}
Nov 03 00:41:52.970850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.970732    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c" event={"ID":"f177d44a-571f-415d-9328-4936506943f5","Type":"ContainerDied","Data":"5bacc85cb9bf76e24167dda0e6414c6797f9c2858104b7c589a6ab36db4f83d7"}
Nov 03 00:41:52.970850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.970750    4550 scope.go:122] "RemoveContainer" containerID="2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac"
Nov 03 00:41:52.982819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.982802    4550 scope.go:122] "RemoveContainer" containerID="2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac"
Nov 03 00:41:52.983102 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:41:52.983086    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac\": container with ID starting with 2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac not found: ID does not exist" containerID="2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac"
Nov 03 00:41:52.983147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:52.983112    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac"} err="failed to get container status \"2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac\": rpc error: code = NotFound desc = could not find container \"2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac\": container with ID starting with 2ab463a542dd2a46257cc95f84ba640ffcec4a7da67542229f1d2f5412cbb7ac not found: ID does not exist"
Nov 03 00:41:53.027870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.027826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:53.028500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.028475    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"]
Nov 03 00:41:53.054969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.054934    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-l2q2c"]
Nov 03 00:41:53.682698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.682663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:53.808977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.808940    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f177d44a-571f-415d-9328-4936506943f5" path="/var/lib/kubelet/pods/f177d44a-571f-415d-9328-4936506943f5/volumes"
Nov 03 00:41:53.901012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.900981    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"]
Nov 03 00:41:53.922330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:53.922298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:54.293979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:54.293944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:54.724281 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:41:54.724247    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:41:55.587295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:55.587259    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:55.687552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:55.687518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:55.787787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:55.787747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:56.524775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:56.524741    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"]
Nov 03 00:41:56.577236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:56.577205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:57.155593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.155544    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:57.642468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.642277    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.666945    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"]
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667816    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f177d44a-571f-415d-9328-4936506943f5" containerName="discovery"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667835    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f177d44a-571f-415d-9328-4936506943f5" containerName="discovery"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667855    4550 cpu_manager.go:414] "containerMap: removing container" podUID="774409d7-ec27-43c9-812c-4687a4632105" containerName="revoke-keys"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667864    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774409d7-ec27-43c9-812c-4687a4632105" containerName="revoke-keys"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667898    4550 cpu_manager.go:414] "containerMap: removing container" podUID="bfc36ba9-924b-4113-9ca7-4a8ed2db6458" containerName="revoke-keys"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667904    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc36ba9-924b-4113-9ca7-4a8ed2db6458" containerName="revoke-keys"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667918    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.667923    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.668014    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc36ba9-924b-4113-9ca7-4a8ed2db6458" containerName="revoke-keys"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.668027    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerName="istio-proxy"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.668033    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f177d44a-571f-415d-9328-4936506943f5" containerName="discovery"
Nov 03 00:41:57.671454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.668039    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="774409d7-ec27-43c9-812c-4687a4632105" containerName="revoke-keys"
Nov 03 00:41:57.672217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.671866    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.718453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.713086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:57.730491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.722743    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"]
Nov 03 00:41:57.730491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.728600    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-d833b8-openshift-default-dockercfg-zn6bv\""
Nov 03 00:41:57.776574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.776527    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.776788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.776581    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.776788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.776607    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.776788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.776639    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.776959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.776806    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.776959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.776924    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.777071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777044    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.777325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777180    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.777325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777198    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.777325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777268    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token\") pod \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\" (UID: \"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9\") "
Nov 03 00:41:57.777574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777462    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.777886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777805    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.777886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777820    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.777886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777846    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.777886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.777846    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:57.778124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778053    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778095    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44g5s\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778128    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778337    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778459    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778494    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778543    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778567    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.778771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778665    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778677    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778686    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778695    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.778771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.778704    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.779978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.779953    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.780125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.780105    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "kube-api-access-4bkjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.780335 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.780311    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:57.780751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.780731    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token" pod "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" (UID: "4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.823657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.823615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:57.850917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.850882    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"]
Nov 03 00:41:57.856767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.856744    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:57.872467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.872403    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fdacdb\""
Nov 03 00:41:57.872467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.872405    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fdacdb-dockercfg-5zjgf\""
Nov 03 00:41:57.872915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.872893    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"]
Nov 03 00:41:57.880191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880159    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880209    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44g5s\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880241    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880324    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880374    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880409    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880489    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880519    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880569    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880601    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880668    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.880971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.880949    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.881338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.881309    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.881667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.881648    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.881754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.881677    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.881754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.881693    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bkjk\" (UniqueName: \"kubernetes.io/projected/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-kube-api-access-4bkjk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.881754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.881708    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.881931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.881791    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.883337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.883315    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.885165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.885137    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.909092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.909044    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"]
Nov 03 00:41:57.909273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.909194    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.911836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.911614    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44g5s\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s\") pod \"e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") " pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:57.915169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.914995    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:57.968453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.965276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:57.973305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.969499    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"]
Nov 03 00:41:58.007462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:57.986603    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4b96\" (UniqueName: \"kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96\") pod \"payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.007805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.007771    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.008486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.008461    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:41:58.096688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.093705    4550 generic.go:356] "Generic (PLEG): container finished" podID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" containerID="64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66" exitCode=0
Nov 03 00:41:58.096688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.093826    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" event={"ID":"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9","Type":"ContainerDied","Data":"64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66"}
Nov 03 00:41:58.096688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.093894    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l" event={"ID":"4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9","Type":"ContainerDied","Data":"4ea25c1b88ed03f24b0aed5fc92d0c9db79cae1f6b4499e585d1ce74605d176f"}
Nov 03 00:41:58.096688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.093945    4550 scope.go:122] "RemoveContainer" containerID="64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66"
Nov 03 00:41:58.096688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.094197    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"
Nov 03 00:41:58.109091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.109057    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4b96\" (UniqueName: \"kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96\") pod \"payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.109237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.109131    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.109237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.109181    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptgz\" (UniqueName: \"kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz\") pod \"payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.109365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.109300    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.110512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.110489    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.129107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.129084    4550 scope.go:122] "RemoveContainer" containerID="64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66"
Nov 03 00:41:58.129399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.129376    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4b96\" (UniqueName: \"kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96\") pod \"payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.129541 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:41:58.129494    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66\": container with ID starting with 64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66 not found: ID does not exist" containerID="64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66"
Nov 03 00:41:58.129593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.129560    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66"} err="failed to get container status \"64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66\": rpc error: code = NotFound desc = could not find container \"64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66\": container with ID starting with 64221a7b63c3e9c93c04cf2c29076265a8c250355f325d9c0cb17d4434adda66 not found: ID does not exist"
Nov 03 00:41:58.152541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.152515    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"]
Nov 03 00:41:58.157554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.157529    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qrj5l"]
Nov 03 00:41:58.171041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.170802    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:58.214320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.211109    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptgz\" (UniqueName: \"kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz\") pod \"payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.214320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.211208    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.214320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.212155    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.233269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.233238    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptgz\" (UniqueName: \"kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz\") pod \"payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.271414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.270675    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:58.423775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.423745    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"]
Nov 03 00:41:58.430057 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:41:58.430013    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df2d376_f3fa_439e_86b5_be2986ea94b4.slice/crio-963ab3f026eb5e77d81855ba6c1580c8b4726a956729c50a816ae4fc4077bcb5 WatchSource:0}: Error finding container 963ab3f026eb5e77d81855ba6c1580c8b4726a956729c50a816ae4fc4077bcb5: Status 404 returned error can't find the container with id 963ab3f026eb5e77d81855ba6c1580c8b4726a956729c50a816ae4fc4077bcb5
Nov 03 00:41:58.433101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.433066    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:58.433182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.433146    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:58.433223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.433184    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:41:58.491317 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:41:58.491271    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c231b8b_a71b_4c4b_9987_b45d6326a61e.slice/crio-f91bdae5baaf0749c55ac703421e6dada87c633e11b6794c8d4c5891fef17284 WatchSource:0}: Error finding container f91bdae5baaf0749c55ac703421e6dada87c633e11b6794c8d4c5891fef17284: Status 404 returned error can't find the container with id f91bdae5baaf0749c55ac703421e6dada87c633e11b6794c8d4c5891fef17284
Nov 03 00:41:58.495874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.495200    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"]
Nov 03 00:41:58.581185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:58.581159    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"]
Nov 03 00:41:58.582860 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:41:58.582822    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493bc4be_10d4_40cc_baf8_04ce4cf49566.slice/crio-aa4cb5204b46ef54894ac956d2e6c48db6b7d5db7805868364358c083b56a0bf WatchSource:0}: Error finding container aa4cb5204b46ef54894ac956d2e6c48db6b7d5db7805868364358c083b56a0bf: Status 404 returned error can't find the container with id aa4cb5204b46ef54894ac956d2e6c48db6b7d5db7805868364358c083b56a0bf
Nov 03 00:41:59.103998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.103955    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" event={"ID":"0c231b8b-a71b-4c4b-9987-b45d6326a61e","Type":"ContainerStarted","Data":"98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7"}
Nov 03 00:41:59.104194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.104027    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:41:59.104194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.104053    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" event={"ID":"0c231b8b-a71b-4c4b-9987-b45d6326a61e","Type":"ContainerStarted","Data":"f91bdae5baaf0749c55ac703421e6dada87c633e11b6794c8d4c5891fef17284"}
Nov 03 00:41:59.105659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.105629    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j" event={"ID":"3df2d376-f3fa-439e-86b5-be2986ea94b4","Type":"ContainerStarted","Data":"5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9"}
Nov 03 00:41:59.105789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.105679    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j" event={"ID":"3df2d376-f3fa-439e-86b5-be2986ea94b4","Type":"ContainerStarted","Data":"963ab3f026eb5e77d81855ba6c1580c8b4726a956729c50a816ae4fc4077bcb5"}
Nov 03 00:41:59.107383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.107350    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" event={"ID":"493bc4be-10d4-40cc-baf8-04ce4cf49566","Type":"ContainerStarted","Data":"a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5"}
Nov 03 00:41:59.107486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.107396    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:41:59.107486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.107411    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" event={"ID":"493bc4be-10d4-40cc-baf8-04ce4cf49566","Type":"ContainerStarted","Data":"aa4cb5204b46ef54894ac956d2e6c48db6b7d5db7805868364358c083b56a0bf"}
Nov 03 00:41:59.145403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.145339    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" podStartSLOduration=2.145326491 podStartE2EDuration="2.145326491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:59.143279067 +0000 UTC m=+2201.464815882" watchObservedRunningTime="2026-11-03 00:41:59.145326491 +0000 UTC m=+2201.466863289"
Nov 03 00:41:59.189374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.189322    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j" podStartSLOduration=2.18931004 podStartE2EDuration="2.18931004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:59.185556296 +0000 UTC m=+2201.507093113" watchObservedRunningTime="2026-11-03 00:41:59.18931004 +0000 UTC m=+2201.510846839"
Nov 03 00:41:59.227752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.227698    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" podStartSLOduration=2.227681492 podStartE2EDuration="2.227681492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:59.224339384 +0000 UTC m=+2201.545876183" watchObservedRunningTime="2026-11-03 00:41:59.227681492 +0000 UTC m=+2201.549218293"
Nov 03 00:41:59.398096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.397624    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:59.599988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.599906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:59.713527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.713485    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:41:59.807835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.807804    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9" path="/var/lib/kubelet/pods/4c7dc7e5-81e8-4233-bb1a-9a0ce2f166d9/volumes"
Nov 03 00:41:59.892211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:41:59.892184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:00.022740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:00.022670    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:42:00.028558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:00.028535    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:42:00.115006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:00.114977    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:42:00.115644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:00.115626    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:42:01.050618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:01.050410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:02.848076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:02.848044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:03.941732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:03.941696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:04.522911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:04.522874    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:05.545527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:05.545491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:05.963862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:05.963823    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"]
Nov 03 00:42:05.969128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:05.969102    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:05.986672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:05.985565    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-955dc3-openshift-default-dockercfg-kvckl\""
Nov 03 00:42:06.005333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.005311    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"]
Nov 03 00:42:06.014390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.014364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:06.116456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116195    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.116456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116283    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.116456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116380    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.116456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116409    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.117004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116467    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.117004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116518    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.117004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.117004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116694    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5cmb\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.117004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.116721    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.219138    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5cmb\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.219765    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220032    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220108    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220179    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220275    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220311    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220346    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220395    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.220501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.220417    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.221785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.221760    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.222209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.222119    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.222385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.222358    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.228353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.228254    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.228353 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.228341    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.231499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.229237    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.241456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.239915    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5cmb\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.244453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.243360    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token\") pod \"e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") " pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.257301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.257105    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:06.287361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.287331    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:06.437455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.436816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:06.602313 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:06.602276    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805b264e_a5ef_4cbc_af02_a7300e4ccaf9.slice/crio-c2fd26e4a3085683e1e641722eb5da032b26e849b4a72f61310fec30909fc4a4 WatchSource:0}: Error finding container c2fd26e4a3085683e1e641722eb5da032b26e849b4a72f61310fec30909fc4a4: Status 404 returned error can't find the container with id c2fd26e4a3085683e1e641722eb5da032b26e849b4a72f61310fec30909fc4a4
Nov 03 00:42:06.604117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.604082    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"]
Nov 03 00:42:06.605530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.605497    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:06.605645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.605564    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:06.605645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:06.605594    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:07.200857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:07.200819    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" event={"ID":"805b264e-a5ef-4cbc-af02-a7300e4ccaf9","Type":"ContainerStarted","Data":"988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"}
Nov 03 00:42:07.201054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:07.200885    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" event={"ID":"805b264e-a5ef-4cbc-af02-a7300e4ccaf9","Type":"ContainerStarted","Data":"c2fd26e4a3085683e1e641722eb5da032b26e849b4a72f61310fec30909fc4a4"}
Nov 03 00:42:07.247559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:07.247514    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podStartSLOduration=2.247500992 podStartE2EDuration="2.247500992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:07.244716364 +0000 UTC m=+2209.566253163" watchObservedRunningTime="2026-11-03 00:42:07.247500992 +0000 UTC m=+2209.569037791"
Nov 03 00:42:07.288466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:07.288420    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:07.295792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:07.295765    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:08.003735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:08.003699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:08.215881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:08.215848    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:08.216742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:08.216719    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:42:08.925022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:08.924981    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv" podUID="76d881f1-4886-42af-924e-8168316a1788" containerName="main" containerID="cri-o://ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460" gracePeriod=45
Nov 03 00:42:09.185532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.185479    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:42:09.223017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.222989    4550 generic.go:356] "Generic (PLEG): container finished" podID="76d881f1-4886-42af-924e-8168316a1788" containerID="ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460" exitCode=0
Nov 03 00:42:09.223152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.223028    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"
Nov 03 00:42:09.223152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.223075    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv" event={"ID":"76d881f1-4886-42af-924e-8168316a1788","Type":"ContainerDied","Data":"ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460"}
Nov 03 00:42:09.223152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.223108    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv" event={"ID":"76d881f1-4886-42af-924e-8168316a1788","Type":"ContainerDied","Data":"005ce7e0496ed65473bd618b8e80f482c25129806ff3473d646f40946b1e29aa"}
Nov 03 00:42:09.223152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.223123    4550 scope.go:122] "RemoveContainer" containerID="ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460"
Nov 03 00:42:09.233152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.233131    4550 scope.go:122] "RemoveContainer" containerID="ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460"
Nov 03 00:42:09.233468 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:09.233427    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460\": container with ID starting with ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460 not found: ID does not exist" containerID="ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460"
Nov 03 00:42:09.233580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.233475    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460"} err="failed to get container status \"ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460\": rpc error: code = NotFound desc = could not find container \"ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460\": container with ID starting with ee35ab86062fd09791302182c6cd1fe3f614d485a2891d7d3ff8f313f69bd460 not found: ID does not exist"
Nov 03 00:42:09.261299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261275    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir\") pod \"76d881f1-4886-42af-924e-8168316a1788\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") "
Nov 03 00:42:09.261413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261329    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs\" (UniqueName: \"kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs\") pod \"76d881f1-4886-42af-924e-8168316a1788\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") "
Nov 03 00:42:09.261413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261369    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm\") pod \"76d881f1-4886-42af-924e-8168316a1788\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") "
Nov 03 00:42:09.261585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261461    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home\") pod \"76d881f1-4886-42af-924e-8168316a1788\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") "
Nov 03 00:42:09.261585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261537    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache\") pod \"76d881f1-4886-42af-924e-8168316a1788\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") "
Nov 03 00:42:09.261585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261553    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir" pod "76d881f1-4886-42af-924e-8168316a1788" (UID: "76d881f1-4886-42af-924e-8168316a1788"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.261747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261637    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh\" (UniqueName: \"kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh\") pod \"76d881f1-4886-42af-924e-8168316a1788\" (UID: \"76d881f1-4886-42af-924e-8168316a1788\") "
Nov 03 00:42:09.261807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.261775    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache" pod "76d881f1-4886-42af-924e-8168316a1788" (UID: "76d881f1-4886-42af-924e-8168316a1788"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.262245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.262219    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.262245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.262246    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.263043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.263020    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home" pod "76d881f1-4886-42af-924e-8168316a1788" (UID: "76d881f1-4886-42af-924e-8168316a1788"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.264605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.264573    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs" pod "76d881f1-4886-42af-924e-8168316a1788" (UID: "76d881f1-4886-42af-924e-8168316a1788"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.264752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.264733    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm" pod "76d881f1-4886-42af-924e-8168316a1788" (UID: "76d881f1-4886-42af-924e-8168316a1788"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.264958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.264936    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh" pod "76d881f1-4886-42af-924e-8168316a1788" (UID: "76d881f1-4886-42af-924e-8168316a1788"). InnerVolumeSpecName "kube-api-access-6jsqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:09.363036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.362999    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jsqh\" (UniqueName: \"kubernetes.io/projected/76d881f1-4886-42af-924e-8168316a1788-kube-api-access-6jsqh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.363036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.363034    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d881f1-4886-42af-924e-8168316a1788-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.363036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.363043    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.363214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.363050    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d881f1-4886-42af-924e-8168316a1788-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.572045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.572013    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"]
Nov 03 00:42:09.583173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.583149    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-e62935/auth-test-model-e62935-kserve-d88cf4789-246pv"]
Nov 03 00:42:09.693778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.693748    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:09.809826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.809746    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d881f1-4886-42af-924e-8168316a1788" path="/var/lib/kubelet/pods/76d881f1-4886-42af-924e-8168316a1788/volumes"
Nov 03 00:42:09.819340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.819314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:09.915921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:09.915888    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:10.118128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:10.118104    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 00:42:10.118282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:10.118180    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 00:42:11.439817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.439782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:11.542300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.542269    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:11.546413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.546386    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" podUID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" containerName="main" containerID="cri-o://44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527" gracePeriod=45
Nov 03 00:42:11.644566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.644542    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"]
Nov 03 00:42:11.645033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.645021    4550 cpu_manager.go:414] "containerMap: removing container" podUID="76d881f1-4886-42af-924e-8168316a1788" containerName="main"
Nov 03 00:42:11.645081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.645034    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d881f1-4886-42af-924e-8168316a1788" containerName="main"
Nov 03 00:42:11.645135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.645126    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d881f1-4886-42af-924e-8168316a1788" containerName="main"
Nov 03 00:42:11.648374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.648357    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:11.669986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.669887    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-955dc3\""
Nov 03 00:42:11.683991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.683970    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-955dc3-dockercfg-8kcn9\""
Nov 03 00:42:11.684981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.684950    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:11.691282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.691228    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"]
Nov 03 00:42:11.840334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.840290    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d\" (UID: \"67072711-766c-4236-9557-8befe2558416\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:11.850474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.845653    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvxk\" (UniqueName: \"kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk\") pod \"payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d\" (UID: \"67072711-766c-4236-9557-8befe2558416\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:11.889997 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.889966    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"]
Nov 03 00:42:11.900141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.900110    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"]
Nov 03 00:42:11.900325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.900224    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:11.950420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.947129    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvxk\" (UniqueName: \"kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk\") pod \"payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d\" (UID: \"67072711-766c-4236-9557-8befe2558416\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:11.950420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.949000    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d\" (UID: \"67072711-766c-4236-9557-8befe2558416\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:11.950420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.950463    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d\" (UID: \"67072711-766c-4236-9557-8befe2558416\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:11.966371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.966265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:11.978539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:11.978507    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvxk\" (UniqueName: \"kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk\") pod \"payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d\" (UID: \"67072711-766c-4236-9557-8befe2558416\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:12.041996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.041972    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:42:12.050348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.050321    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lxng\" (UniqueName: \"kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng\") pod \"payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.050556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.050532    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.151945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.151914    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir\") pod \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") "
Nov 03 00:42:12.152112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.151997    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home\") pod \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") "
Nov 03 00:42:12.152112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.152066    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm\") pod \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") "
Nov 03 00:42:12.152235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.152121    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9\" (UniqueName: \"kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9\") pod \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") "
Nov 03 00:42:12.152235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.152201    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir" pod "4cdd1bb1-e415-47fc-94cd-f49d60726c25" (UID: "4cdd1bb1-e415-47fc-94cd-f49d60726c25"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:12.152488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.152461    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home" pod "4cdd1bb1-e415-47fc-94cd-f49d60726c25" (UID: "4cdd1bb1-e415-47fc-94cd-f49d60726c25"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:12.152674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.152655    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs\") pod \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") "
Nov 03 00:42:12.152854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.152833    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache\") pod \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\" (UID: \"4cdd1bb1-e415-47fc-94cd-f49d60726c25\") "
Nov 03 00:42:12.153262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.153235    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.153362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.153275    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache" pod "4cdd1bb1-e415-47fc-94cd-f49d60726c25" (UID: "4cdd1bb1-e415-47fc-94cd-f49d60726c25"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:12.153511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.153483    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lxng\" (UniqueName: \"kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng\") pod \"payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.153649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.153632    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:12.153718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.153654    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:12.153718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.153668    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:12.154024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.154003    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.154639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.154612    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm" pod "4cdd1bb1-e415-47fc-94cd-f49d60726c25" (UID: "4cdd1bb1-e415-47fc-94cd-f49d60726c25"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:12.154729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.154656    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9" pod "4cdd1bb1-e415-47fc-94cd-f49d60726c25" (UID: "4cdd1bb1-e415-47fc-94cd-f49d60726c25"). InnerVolumeSpecName "kube-api-access-v8nt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:12.155124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.155104    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs" pod "4cdd1bb1-e415-47fc-94cd-f49d60726c25" (UID: "4cdd1bb1-e415-47fc-94cd-f49d60726c25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:12.176996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.176969    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lxng\" (UniqueName: \"kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng\") pod \"payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.225072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.225018    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:12.255056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.255031    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cdd1bb1-e415-47fc-94cd-f49d60726c25-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:12.255155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.255058    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8nt9\" (UniqueName: \"kubernetes.io/projected/4cdd1bb1-e415-47fc-94cd-f49d60726c25-kube-api-access-v8nt9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:12.255155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.255069    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cdd1bb1-e415-47fc-94cd-f49d60726c25-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:12.264511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.264493    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:12.412219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.412173    4550 generic.go:356] "Generic (PLEG): container finished" podID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" containerID="44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527" exitCode=0
Nov 03 00:42:12.412593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.412566    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" event={"ID":"4cdd1bb1-e415-47fc-94cd-f49d60726c25","Type":"ContainerDied","Data":"44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527"}
Nov 03 00:42:12.412778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.412759    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" event={"ID":"4cdd1bb1-e415-47fc-94cd-f49d60726c25","Type":"ContainerDied","Data":"6deb75326df54e966a0e3fe690b6101c67d9c9b6a51e9a5fa13b13e5ccd11cff"}
Nov 03 00:42:12.412903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.412889    4550 scope.go:122] "RemoveContainer" containerID="44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527"
Nov 03 00:42:12.413128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.413111    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"
Nov 03 00:42:12.455756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.452374    4550 scope.go:122] "RemoveContainer" containerID="44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527"
Nov 03 00:42:12.455756 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:12.452745    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527\": container with ID starting with 44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527 not found: ID does not exist" containerID="44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527"
Nov 03 00:42:12.455756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.452778    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527"} err="failed to get container status \"44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527\": rpc error: code = NotFound desc = could not find container \"44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527\": container with ID starting with 44bd32ec46c300e9966496be1d9bf724fed1793706a2ab53d5fd8e0b7ec39527 not found: ID does not exist"
Nov 03 00:42:12.494536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.486802    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"]
Nov 03 00:42:12.496676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.496651    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp"]
Nov 03 00:42:12.534689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.534635    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"]
Nov 03 00:42:12.765819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:12.764516    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"]
Nov 03 00:42:13.028968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.028885    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-20109d/auth-test-model-20109d-kserve-bbfb4869-bssjp" podUID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" containerName="main" probeResult="failure" output="Get \"https://10.133.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:42:13.420478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.420440    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" event={"ID":"67072711-766c-4236-9557-8befe2558416","Type":"ContainerStarted","Data":"8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604"}
Nov 03 00:42:13.420646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.420497    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" event={"ID":"67072711-766c-4236-9557-8befe2558416","Type":"ContainerStarted","Data":"ada0e84a8df0c951697eacfebadcaac58975810e13caec570ac4e9fc5679aa5f"}
Nov 03 00:42:13.420646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.420541    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:13.423031 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.423007    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" event={"ID":"99f13630-6b7e-4799-bc76-20e7d84fef3b","Type":"ContainerStarted","Data":"5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6"}
Nov 03 00:42:13.423162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.423043    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" event={"ID":"99f13630-6b7e-4799-bc76-20e7d84fef3b","Type":"ContainerStarted","Data":"e11f662161bc7d2b53e57698fe04b00171967ae8cf70534aaf49db68fbccfc90"}
Nov 03 00:42:13.423162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.423077    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:13.465105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.465069    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" podStartSLOduration=2.465059152 podStartE2EDuration="2.465059152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:13.462492414 +0000 UTC m=+2215.784029212" watchObservedRunningTime="2026-11-03 00:42:13.465059152 +0000 UTC m=+2215.786595969"
Nov 03 00:42:13.511285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.511244    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" podStartSLOduration=2.511232896 podStartE2EDuration="2.511232896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:13.509020927 +0000 UTC m=+2215.830557725" watchObservedRunningTime="2026-11-03 00:42:13.511232896 +0000 UTC m=+2215.832769694"
Nov 03 00:42:13.808720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.808647    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" path="/var/lib/kubelet/pods/4cdd1bb1-e415-47fc-94cd-f49d60726c25/volumes"
Nov 03 00:42:13.816687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:13.816655    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:15.767655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:15.767615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:16.883074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.883040    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"]
Nov 03 00:42:16.883757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.883735    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" containerName="main"
Nov 03 00:42:16.883846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.883758    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" containerName="main"
Nov 03 00:42:16.883929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.883917    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cdd1bb1-e415-47fc-94cd-f49d60726c25" containerName="main"
Nov 03 00:42:16.887779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.887757    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:16.906328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.906300    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"]
Nov 03 00:42:16.915342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.915319    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d833b8\"/\"openshift-service-ca.crt\""
Nov 03 00:42:16.915478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.915329    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d833b8\"/\"kube-root-ca.crt\""
Nov 03 00:42:16.921843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.917177    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d833b8\"/\"default-dockercfg-m6jw5\""
Nov 03 00:42:16.921843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.917193    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d833b8\"/\"noten-model-d833b8-kserve-self-signed-certs\""
Nov 03 00:42:16.924397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.924375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:16.961926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.961895    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:16.962038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.961956    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcmgd\" (UniqueName: \"kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:16.962038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.961988    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:16.962189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.962114    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:16.962189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.962177    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:16.962300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:16.962207    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.063528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.063489    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.063729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.063551    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcmgd\" (UniqueName: \"kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.063729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.063581    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.063729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.063716    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.063954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.063797    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.063954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.063831    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.064307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.064279    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.064406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.064279    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.064548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.064524    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.066845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.066824    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.070911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.070890    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.109456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.107808    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcmgd\" (UniqueName: \"kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd\") pod \"noten-model-d833b8-kserve-7c6fff677c-64lcd\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") " pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.203455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.200808    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:17.522890 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:17.522847    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5d5a9c_0ac9_4b42_b170_02ca4730aee4.slice/crio-37b63a22ceb53e1ff3190bcf4c3e1ff9ebfdc168fbadc74dd26c002275f2b59f WatchSource:0}: Error finding container 37b63a22ceb53e1ff3190bcf4c3e1ff9ebfdc168fbadc74dd26c002275f2b59f: Status 404 returned error can't find the container with id 37b63a22ceb53e1ff3190bcf4c3e1ff9ebfdc168fbadc74dd26c002275f2b59f
Nov 03 00:42:17.524878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.524853    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"]
Nov 03 00:42:17.822537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:17.822463    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:18.454376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:18.454343    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd" event={"ID":"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4","Type":"ContainerStarted","Data":"b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db"}
Nov 03 00:42:18.454748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:18.454386    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd" event={"ID":"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4","Type":"ContainerStarted","Data":"37b63a22ceb53e1ff3190bcf4c3e1ff9ebfdc168fbadc74dd26c002275f2b59f"}
Nov 03 00:42:18.494875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:18.494817    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd" podStartSLOduration=2.494799757 podStartE2EDuration="2.494799757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:18.493126304 +0000 UTC m=+2220.814663102" watchObservedRunningTime="2026-11-03 00:42:18.494799757 +0000 UTC m=+2220.816336559"
Nov 03 00:42:19.201381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:19.201348    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:19.649231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:19.649158    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:21.427271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:21.427233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:23.000584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:23.000547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:24.376105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.376069    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"]
Nov 03 00:42:24.409382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.409341    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:24.433120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.433092    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 00:42:24.433273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.433165    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 00:42:24.573147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.573085    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"]
Nov 03 00:42:24.573635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.573605    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j" podUID="3df2d376-f3fa-439e-86b5-be2986ea94b4" containerName="istio-proxy" containerID="cri-o://5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9" gracePeriod=30
Nov 03 00:42:24.576898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.576360    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:24.768135 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:24.766933    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:42:24.936019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:24.935981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:25.095636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:25.095552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:25.213545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:25.213508    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:25.319062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:25.319027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:25.461415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:25.461365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:27.580729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:27.580694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:29.550252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:29.550217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:29.889647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:29.889621    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:42:30.032855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.032820    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.032874    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.032978    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033010    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033093    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033154    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033208    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033241    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.033369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033313    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs\") pod \"3df2d376-f3fa-439e-86b5-be2986ea94b4\" (UID: \"3df2d376-f3fa-439e-86b5-be2986ea94b4\") "
Nov 03 00:42:30.034124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033576    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:42:30.034124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033888    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:30.034124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.033942    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3df2d376-f3fa-439e-86b5-be2986ea94b4-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.034124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.034052    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:30.034124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.034102    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:30.034747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.034724    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:30.035339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.035300    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:42:30.035676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.035629    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "kube-api-access-44g5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:30.035676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.035660    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:30.035898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.035878    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token" pod "3df2d376-f3fa-439e-86b5-be2986ea94b4" (UID: "3df2d376-f3fa-439e-86b5-be2986ea94b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135691    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44g5s\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-kube-api-access-44g5s\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135725    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135738    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135749    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135761    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135771    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135781    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.135790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.135792    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3df2d376-f3fa-439e-86b5-be2986ea94b4-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:30.159267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.159229    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:30.601951 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.601916    4550 generic.go:356] "Generic (PLEG): container finished" podID="3df2d376-f3fa-439e-86b5-be2986ea94b4" containerID="5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9" exitCode=0
Nov 03 00:42:30.602372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.601998    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"
Nov 03 00:42:30.602372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.602005    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j" event={"ID":"3df2d376-f3fa-439e-86b5-be2986ea94b4","Type":"ContainerDied","Data":"5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9"}
Nov 03 00:42:30.602372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.602090    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j" event={"ID":"3df2d376-f3fa-439e-86b5-be2986ea94b4","Type":"ContainerDied","Data":"963ab3f026eb5e77d81855ba6c1580c8b4726a956729c50a816ae4fc4077bcb5"}
Nov 03 00:42:30.602372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.602113    4550 scope.go:122] "RemoveContainer" containerID="5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9"
Nov 03 00:42:30.613885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.613866    4550 scope.go:122] "RemoveContainer" containerID="5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9"
Nov 03 00:42:30.614138 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:30.614111    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9\": container with ID starting with 5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9 not found: ID does not exist" containerID="5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9"
Nov 03 00:42:30.614208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.614145    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9"} err="failed to get container status \"5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9\": rpc error: code = NotFound desc = could not find container \"5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9\": container with ID starting with 5e3f1af662d723d1d84bd3930a8ffa215ee4f4afd3ad0b7619ac1bb249ddc6f9 not found: ID does not exist"
Nov 03 00:42:30.671841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.671818    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"]
Nov 03 00:42:30.684595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:30.684575    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-d833b8-openshift-default-74644d6bd9-vt69j"]
Nov 03 00:42:31.453218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:31.453190    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:31.808657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:31.808582    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3df2d376-f3fa-439e-86b5-be2986ea94b4" path="/var/lib/kubelet/pods/3df2d376-f3fa-439e-86b5-be2986ea94b4/volumes"
Nov 03 00:42:33.255128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:33.255094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:34.106268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:34.106232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:35.178073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.178036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:35.479793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.479707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:35.883768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.883680    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"]
Nov 03 00:42:35.884727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.884703    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3df2d376-f3fa-439e-86b5-be2986ea94b4" containerName="istio-proxy"
Nov 03 00:42:35.884867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.884854    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3df2d376-f3fa-439e-86b5-be2986ea94b4" containerName="istio-proxy"
Nov 03 00:42:35.885224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.885208    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3df2d376-f3fa-439e-86b5-be2986ea94b4" containerName="istio-proxy"
Nov 03 00:42:35.890646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.890623    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:35.919138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.916420    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"]
Nov 03 00:42:35.921180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.921127    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d4d8c2-openshift-default-dockercfg-nskw8\""
Nov 03 00:42:35.934554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:35.934533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:36.018979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.018943    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019019    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019058    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019085    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019159    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58zj\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019179    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019313    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019421    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.019537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.019478    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.050411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.050382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133221    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133282    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133446    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133473    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133510    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133539    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133598    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58zj\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133630    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.133714    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.134181    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.135077    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.136579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.136323    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.137397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.136806    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.137397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.137009    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.139738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.139714    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.140400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.140361    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.159464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.159423    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.160311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.160240    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58zj\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj\") pod \"e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") " pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.218972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.218677    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:36.555105 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:36.555062    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dff03b7_1b77_43f5_9204_29b75194a7d9.slice/crio-e072fa296edab86b47ea55e1a8395534babbef4296b785640f2a1fce98cf4712 WatchSource:0}: Error finding container e072fa296edab86b47ea55e1a8395534babbef4296b785640f2a1fce98cf4712: Status 404 returned error can't find the container with id e072fa296edab86b47ea55e1a8395534babbef4296b785640f2a1fce98cf4712
Nov 03 00:42:36.558900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.558863    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:36.559017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.558950    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:36.559017 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.558996    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:36.559588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.559369    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"]
Nov 03 00:42:36.650254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.650217    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" event={"ID":"7dff03b7-1b77-43f5-9204-29b75194a7d9","Type":"ContainerStarted","Data":"0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"}
Nov 03 00:42:36.650410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.650286    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" event={"ID":"7dff03b7-1b77-43f5-9204-29b75194a7d9","Type":"ContainerStarted","Data":"e072fa296edab86b47ea55e1a8395534babbef4296b785640f2a1fce98cf4712"}
Nov 03 00:42:36.694093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.693622    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podStartSLOduration=1.693605964 podStartE2EDuration="1.693605964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:36.692423228 +0000 UTC m=+2239.013960049" watchObservedRunningTime="2026-11-03 00:42:36.693605964 +0000 UTC m=+2239.015142764"
Nov 03 00:42:36.853026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:36.852937    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:37.177463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:37.174809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:37.219842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:37.219811    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:37.235830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:37.235797    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:37.656509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:37.656481    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:37.658032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:37.658006    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:42:38.705857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:38.705826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:39.129259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.129173    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:39.237603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.237569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:39.414563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.414531    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd" podUID="cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" containerName="main" containerID="cri-o://b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db" gracePeriod=45
Nov 03 00:42:39.442576 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:39.442550    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5d5a9c_0ac9_4b42_b170_02ca4730aee4.slice/crio-b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5d5a9c_0ac9_4b42_b170_02ca4730aee4.slice/crio-conmon-b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:42:39.442761 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:39.442739    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5d5a9c_0ac9_4b42_b170_02ca4730aee4.slice/crio-conmon-b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:42:39.620748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.620645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:39.672709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.672629    4550 generic.go:356] "Generic (PLEG): container finished" podID="cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" containerID="b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db" exitCode=0
Nov 03 00:42:39.672843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.672704    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd" event={"ID":"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4","Type":"ContainerDied","Data":"b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db"}
Nov 03 00:42:39.716512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.716489    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:39.765916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.765637    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs\") pod \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") "
Nov 03 00:42:39.765916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.765676    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache\") pod \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") "
Nov 03 00:42:39.766114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.766090    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir\") pod \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") "
Nov 03 00:42:39.766685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.766234    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd\" (UniqueName: \"kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd\") pod \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") "
Nov 03 00:42:39.766685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.766297    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm\") pod \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") "
Nov 03 00:42:39.766685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.766329    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home\") pod \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\" (UID: \"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4\") "
Nov 03 00:42:39.769121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.769074    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache" pod "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" (UID: "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:39.770371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.770347    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:39.770521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.770450    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir" pod "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" (UID: "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:39.770673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.770641    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm" pod "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" (UID: "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:39.771106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.771077    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs" pod "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" (UID: "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:39.772062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.772040    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home" pod "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" (UID: "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:39.772302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.772275    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd" pod "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" (UID: "cb5d5a9c-0ac9-4b42-b170-02ca4730aee4"). InnerVolumeSpecName "kube-api-access-xcmgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:39.871653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.871626    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:39.871841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.871816    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:39.871841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.871839    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:39.871988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.871854    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:39.871988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:39.871868    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcmgd\" (UniqueName: \"kubernetes.io/projected/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4-kube-api-access-xcmgd\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:42:40.047837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.047760    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"]
Nov 03 00:42:40.048537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.048522    4550 cpu_manager.go:414] "containerMap: removing container" podUID="cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" containerName="main"
Nov 03 00:42:40.048613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.048539    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" containerName="main"
Nov 03 00:42:40.048656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.048647    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" containerName="main"
Nov 03 00:42:40.052218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.052196    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.072688    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c738e0-openshift-default-dockercfg-kqfjn\""
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.073898    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pjf\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.073964    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074000    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074115    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074179    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074203    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074259    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.074319    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.078773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.077382    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"]
Nov 03 00:42:40.109061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.109030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183195    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183249    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183317    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183377    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183448    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pjf\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183527    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183569    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183623    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.183678    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.184168    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.193456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.184414    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.213419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.207584    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.213419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.208285    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.213419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.208583    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.223449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.214116    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.232233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.232208    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.245515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.240717    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.245515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.245162    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pjf\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf\") pod \"e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") " pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.369793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.369698    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:40.461291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.461052    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:40.684100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.684060    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd" event={"ID":"cb5d5a9c-0ac9-4b42-b170-02ca4730aee4","Type":"ContainerDied","Data":"37b63a22ceb53e1ff3190bcf4c3e1ff9ebfdc168fbadc74dd26c002275f2b59f"}
Nov 03 00:42:40.684291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.684120    4550 scope.go:122] "RemoveContainer" containerID="b239c6f894e7d71a5e1eb387c2c13254c4082ed74fe5cbda87fc3df1aa63f1db"
Nov 03 00:42:40.684291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.684280    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"
Nov 03 00:42:40.698640 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:40.694982    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91526e27_ec44_4306_aed5_67fb95bad247.slice/crio-1abcfeba87eb05e55f9322fdcea1ed7f3064516fc5f966ed3842e9ad1edc4622 WatchSource:0}: Error finding container 1abcfeba87eb05e55f9322fdcea1ed7f3064516fc5f966ed3842e9ad1edc4622: Status 404 returned error can't find the container with id 1abcfeba87eb05e55f9322fdcea1ed7f3064516fc5f966ed3842e9ad1edc4622
Nov 03 00:42:40.698640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.696047    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"]
Nov 03 00:42:40.701982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.701952    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:40.702088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.702021    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:40.702088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.702052    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:42:40.722915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.722889    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:40.740208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.740185    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"]
Nov 03 00:42:40.745010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.744988    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d833b8/noten-model-d833b8-kserve-7c6fff677c-64lcd"]
Nov 03 00:42:40.836216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.836136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:40.924818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.924787    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"]
Nov 03 00:42:40.928841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.928812    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:40.939369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.939348    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:40.947856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.947831    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d4d8c2\""
Nov 03 00:42:40.948068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:40.948050    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"]
Nov 03 00:42:41.017381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.017337    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"]
Nov 03 00:42:41.024312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.024269    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"]
Nov 03 00:42:41.024505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.024378    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.040470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.040408    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d4d8c2-dockercfg-wzrml\""
Nov 03 00:42:41.115689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.115641    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.116660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.116633    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f8pr\" (UniqueName: \"kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr\") pod \"payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.239510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.237071    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjhw\" (UniqueName: \"kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw\") pod \"payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.239510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.237238    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.239510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.237306    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.239510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.237369    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8pr\" (UniqueName: \"kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr\") pod \"payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.250450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.246868    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.275256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.275166    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8pr\" (UniqueName: \"kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr\") pod \"payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.347115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.347083    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.347273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.347194    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjhw\" (UniqueName: \"kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw\") pod \"payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.347767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.347749    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.367447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.367406    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjhw\" (UniqueName: \"kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw\") pod \"payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.543668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.543595    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:41.652542 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.652514    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:41.702641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.702607    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" event={"ID":"91526e27-ec44-4306-aed5-67fb95bad247","Type":"ContainerStarted","Data":"e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"}
Nov 03 00:42:41.702761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.702685    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" event={"ID":"91526e27-ec44-4306-aed5-67fb95bad247","Type":"ContainerStarted","Data":"1abcfeba87eb05e55f9322fdcea1ed7f3064516fc5f966ed3842e9ad1edc4622"}
Nov 03 00:42:41.754730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.754687    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podStartSLOduration=1.754672217 podStartE2EDuration="1.754672217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:41.751616214 +0000 UTC m=+2244.073153014" watchObservedRunningTime="2026-11-03 00:42:41.754672217 +0000 UTC m=+2244.076209015"
Nov 03 00:42:41.811876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.811759    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb5d5a9c-0ac9-4b42-b170-02ca4730aee4" path="/var/lib/kubelet/pods/cb5d5a9c-0ac9-4b42-b170-02ca4730aee4/volumes"
Nov 03 00:42:41.869455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:41.861088    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"]
Nov 03 00:42:41.869455 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:41.861579    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd0c6ce_8286_40af_b06f_bbcfd8f749e7.slice/crio-16e7f1eaca27eee1897577e819df1d5258fec207e87841e2707cd227d97ab8d8 WatchSource:0}: Error finding container 16e7f1eaca27eee1897577e819df1d5258fec207e87841e2707cd227d97ab8d8: Status 404 returned error can't find the container with id 16e7f1eaca27eee1897577e819df1d5258fec207e87841e2707cd227d97ab8d8
Nov 03 00:42:42.012638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.012605    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"]
Nov 03 00:42:42.370027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.369960    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:42.376643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.376618    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:42.419960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.419936    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:42.711174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.711138    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" event={"ID":"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7","Type":"ContainerStarted","Data":"47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8"}
Nov 03 00:42:42.711352 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.711203    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" event={"ID":"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7","Type":"ContainerStarted","Data":"16e7f1eaca27eee1897577e819df1d5258fec207e87841e2707cd227d97ab8d8"}
Nov 03 00:42:42.711352 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.711237    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:42.712749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.712722    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" event={"ID":"4f2cf1ee-59d3-457c-b737-b672bc49eb58","Type":"ContainerStarted","Data":"b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93"}
Nov 03 00:42:42.712877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.712771    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" event={"ID":"4f2cf1ee-59d3-457c-b737-b672bc49eb58","Type":"ContainerStarted","Data":"70c80e44aeecb325a5d798826e486e5d0ab85046e402d5d8cd3d966adc375d94"}
Nov 03 00:42:42.712938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.712900    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:42.712998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.712941    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:42.713676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.713658    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:42:42.754387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.754348    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" podStartSLOduration=2.754336665 podStartE2EDuration="2.754336665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:42.751616864 +0000 UTC m=+2245.073153660" watchObservedRunningTime="2026-11-03 00:42:42.754336665 +0000 UTC m=+2245.075873464"
Nov 03 00:42:42.839500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:42.837127    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" podStartSLOduration=2.837108059 podStartE2EDuration="2.837108059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:42.833402791 +0000 UTC m=+2245.154939615" watchObservedRunningTime="2026-11-03 00:42:42.837108059 +0000 UTC m=+2245.158644858"
Nov 03 00:42:44.208795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:44.208762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:44.346543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:44.346511    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:45.870426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:45.870382    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:45.983718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:45.983683    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"]
Nov 03 00:42:45.985914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:45.985878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:45.988491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:45.988469    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.007371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.007345    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"]
Nov 03 00:42:46.015500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.015388    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c738e0\""
Nov 03 00:42:46.015500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.015464    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c738e0-dockercfg-m8qlr\""
Nov 03 00:42:46.021685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.021661    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgr8\" (UniqueName: \"kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8\") pod \"payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.021964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.021945    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.039332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.039307    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"]
Nov 03 00:42:46.043202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.043186    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.063071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.063048    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"]
Nov 03 00:42:46.233455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.229448    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddx9\" (UniqueName: \"kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9\") pod \"payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.233455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.229645    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.233455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.229928    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgr8\" (UniqueName: \"kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8\") pod \"payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.233455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.230421    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.233455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.231283    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.262192 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.256595    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgr8\" (UniqueName: \"kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8\") pod \"payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.304691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.304654    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.334408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.334370    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.338088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.338060    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddx9\" (UniqueName: \"kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9\") pod \"payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.339355 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.339327    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.371789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.371526    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:46.375100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.375072    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddx9\" (UniqueName: \"kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9\") pod \"payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.468590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.468558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:46.570121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.570021    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:46.632083 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:46.632043    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0f087f_fb65_4426_a283_bbb296179bf7.slice/crio-a1205aca422ae46b0246110ede77d472afabe39faaaa1e3a3761f65f39f293ed WatchSource:0}: Error finding container a1205aca422ae46b0246110ede77d472afabe39faaaa1e3a3761f65f39f293ed: Status 404 returned error can't find the container with id a1205aca422ae46b0246110ede77d472afabe39faaaa1e3a3761f65f39f293ed
Nov 03 00:42:46.633628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.633597    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"]
Nov 03 00:42:46.656270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.656237    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:46.740098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.738838    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" event={"ID":"2e0f087f-fb65-4426-a283-bbb296179bf7","Type":"ContainerStarted","Data":"80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018"}
Nov 03 00:42:46.740098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.738907    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" event={"ID":"2e0f087f-fb65-4426-a283-bbb296179bf7","Type":"ContainerStarted","Data":"a1205aca422ae46b0246110ede77d472afabe39faaaa1e3a3761f65f39f293ed"}
Nov 03 00:42:46.740098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.739964    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:46.788758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.788703    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" podStartSLOduration=1.788689513 podStartE2EDuration="1.788689513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:46.784654792 +0000 UTC m=+2249.106191636" watchObservedRunningTime="2026-11-03 00:42:46.788689513 +0000 UTC m=+2249.110226308"
Nov 03 00:42:46.972094 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:42:46.972064    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f77735_43c7_4784_8e9e_87cd68607eca.slice/crio-3331568a742a7564cf7af29e59fe04262208375911851215889d17d6c73c80b9 WatchSource:0}: Error finding container 3331568a742a7564cf7af29e59fe04262208375911851215889d17d6c73c80b9: Status 404 returned error can't find the container with id 3331568a742a7564cf7af29e59fe04262208375911851215889d17d6c73c80b9
Nov 03 00:42:46.974288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:46.974159    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"]
Nov 03 00:42:47.747225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:47.747185    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" event={"ID":"64f77735-43c7-4784-8e9e-87cd68607eca","Type":"ContainerStarted","Data":"e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497"}
Nov 03 00:42:47.747225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:47.747243    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" event={"ID":"64f77735-43c7-4784-8e9e-87cd68607eca","Type":"ContainerStarted","Data":"3331568a742a7564cf7af29e59fe04262208375911851215889d17d6c73c80b9"}
Nov 03 00:42:47.747511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:47.747450    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:42:47.792189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:47.792119    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" podStartSLOduration=1.792098867 podStartE2EDuration="1.792098867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:47.787572566 +0000 UTC m=+2250.109109365" watchObservedRunningTime="2026-11-03 00:42:47.792098867 +0000 UTC m=+2250.113635693"
Nov 03 00:42:48.375782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:48.375742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:50.123769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:50.123732    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:51.803890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:51.803858    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:53.528136 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:53.528097    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:53.721889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:53.721856    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 00:42:53.722105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:53.721936    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 00:42:54.725836 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:42:54.725795    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:42:55.220047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:55.220015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:56.999525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:56.999480    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:58.644848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:58.644808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:42:58.757468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:58.757403    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 00:42:58.757647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:42:58.757502    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 00:43:00.179939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:00.179899    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:01.829176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:01.829129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:03.600939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:03.600900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:04.253915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:04.253871    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:05.432975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:05.432939    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:05.543584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:05.543550    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:05.842571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:05.842477    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:05.939350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:05.939309    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:06.042393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:06.042359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:06.246501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:06.246462    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:06.348276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:06.348243    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:06.851456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:06.851407    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:06.946594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:06.946568    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:07.389329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:07.389241    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:08.071456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.070469    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:08.195444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.195385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:08.248280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.248247    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"]
Nov 03 00:43:08.253408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.253391    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.286459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.272997    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-2274efc91ed5\""
Nov 03 00:43:08.286459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.285788    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"openshift-service-ca.crt\""
Nov 03 00:43:08.286459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.285932    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-e2e--2e23f326-dockercfg-njdlx\""
Nov 03 00:43:08.286794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.286639    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"kube-root-ca.crt\""
Nov 03 00:43:08.286794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.286714    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdacdb\""
Nov 03 00:43:08.286794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.286721    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"routing-overlay\""
Nov 03 00:43:08.289688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.289656    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"]
Nov 03 00:43:08.411783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.410166    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cmp\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.411783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.411454    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.411783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.411521    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.411783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.411551    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.519455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.519306    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.519455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.519377    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.519455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.519408    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.519803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.519585    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cmp\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.521478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.520944    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.522110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.522057    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.529832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.529802    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.549195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.549159    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cmp\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.593721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.593684    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:08.921115 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:43:08.921085    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18de8fc3_d694_4f27_935d_bef519be6005.slice/crio-c4887d3e36dc3568ca72872a67e4b20e3cebede9936f6874e95df11f5383fa08 WatchSource:0}: Error finding container c4887d3e36dc3568ca72872a67e4b20e3cebede9936f6874e95df11f5383fa08: Status 404 returned error can't find the container with id c4887d3e36dc3568ca72872a67e4b20e3cebede9936f6874e95df11f5383fa08
Nov 03 00:43:08.923215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:08.923194    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"]
Nov 03 00:43:09.209143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:09.209045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:09.836451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:09.836399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:09.921625 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:09.921589    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7" event={"ID":"18de8fc3-d694-4f27-935d-bef519be6005","Type":"ContainerStarted","Data":"46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef"}
Nov 03 00:43:09.921840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:09.921648    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7" event={"ID":"18de8fc3-d694-4f27-935d-bef519be6005","Type":"ContainerStarted","Data":"c4887d3e36dc3568ca72872a67e4b20e3cebede9936f6874e95df11f5383fa08"}
Nov 03 00:43:09.921840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:09.921733    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:09.982976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:09.982909    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7" podStartSLOduration=1.9828892919999999 podStartE2EDuration="1.982889292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:09.979734023 +0000 UTC m=+2272.301270847" watchObservedRunningTime="2026-11-03 00:43:09.982889292 +0000 UTC m=+2272.304426095"
Nov 03 00:43:10.387512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.386615    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"]
Nov 03 00:43:10.411212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.411181    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.425987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.425622    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"]
Nov 03 00:43:10.436464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.430771    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-850433-openshift-default-dockercfg-qt766\""
Nov 03 00:43:10.436464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.431796    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:10.473486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.455188    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.473851    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.473921    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.473983    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.474016    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.474100    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8nxb\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.474171    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.474237    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.482461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.474268    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635000    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635107    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635158    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635209    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635246    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635318    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8nxb\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635386    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635519    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.635560    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.636619    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.636942    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.637177    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.637597    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.638633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.637882    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.639464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.639358    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.680066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.675512    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.699866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.699833    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8nxb\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.703450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.701729    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:10.832180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.832141    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:10.937613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.937570    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:10.946689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:10.946655    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:11.331097 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:43:11.331063    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aeed2a_e1e5_4546_93bd_94db6c9953ae.slice/crio-622c7d83150183e89294574f8df99e6dc6b313f28feddd58257076ead93bbd0d WatchSource:0}: Error finding container 622c7d83150183e89294574f8df99e6dc6b313f28feddd58257076ead93bbd0d: Status 404 returned error can't find the container with id 622c7d83150183e89294574f8df99e6dc6b313f28feddd58257076ead93bbd0d
Nov 03 00:43:11.334184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.334153    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"]
Nov 03 00:43:11.334463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.334411    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:11.334561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.334496    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:11.334561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.334525    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:11.779038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.779007    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:11.941931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.941888    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" event={"ID":"48aeed2a-e1e5-4546-93bd-94db6c9953ae","Type":"ContainerStarted","Data":"f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"}
Nov 03 00:43:11.942138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.941960    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" event={"ID":"48aeed2a-e1e5-4546-93bd-94db6c9953ae","Type":"ContainerStarted","Data":"622c7d83150183e89294574f8df99e6dc6b313f28feddd58257076ead93bbd0d"}
Nov 03 00:43:11.947237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:11.947212    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:12.011074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:12.011007    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podStartSLOduration=2.010986738 podStartE2EDuration="2.010986738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:11.99696413 +0000 UTC m=+2274.318500929" watchObservedRunningTime="2026-11-03 00:43:12.010986738 +0000 UTC m=+2274.332523640"
Nov 03 00:43:12.131969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:12.131879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:12.636042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:12.636003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:12.955328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:12.955296    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:13.955937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:13.955901    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:13.956926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:13.956907    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:43:14.414768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:14.413472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:14.509966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:14.509927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:14.623770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:14.623733    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:14.733253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:14.733130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:15.983626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:15.983586    4550 generic.go:356] "Generic (PLEG): container finished" podID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerID="988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b" exitCode=0
Nov 03 00:43:15.984111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:15.983661    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" event={"ID":"805b264e-a5ef-4cbc-af02-a7300e4ccaf9","Type":"ContainerDied","Data":"988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"}
Nov 03 00:43:15.984254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:15.984237    4550 scope.go:122] "RemoveContainer" containerID="988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"
Nov 03 00:43:15.985965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:15.985933    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:15.986093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:15.986020    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:15.986093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:15.986067    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:16.223505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.223463    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:16.334169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.334073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:16.414035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.414000    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"]
Nov 03 00:43:16.418299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.418279    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.431215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.431184    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"]
Nov 03 00:43:16.544455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.542038    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.544455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.542089    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxh4\" (UniqueName: \"kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4\") pod \"payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.562457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.557829    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:16.562457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.562142    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-850433\""
Nov 03 00:43:16.562457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.562382    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-850433-dockercfg-bj5ph\""
Nov 03 00:43:16.669110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.658117    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.672730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.672143    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxh4\" (UniqueName: \"kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4\") pod \"payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.673115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.671359    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.713154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.713118    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxh4\" (UniqueName: \"kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4\") pod \"payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.739802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.739765    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:16.997804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:16.997714    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" event={"ID":"805b264e-a5ef-4cbc-af02-a7300e4ccaf9","Type":"ContainerStarted","Data":"0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a"}
Nov 03 00:43:17.043225 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:43:17.043170    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658355d0_4279_4077_9d55_f38b8fb7d19b.slice/crio-26722ddd479c4cea0f75c3c3d3213539be5b22a7615ebe86532dcfaf5a4bf3f7 WatchSource:0}: Error finding container 26722ddd479c4cea0f75c3c3d3213539be5b22a7615ebe86532dcfaf5a4bf3f7: Status 404 returned error can't find the container with id 26722ddd479c4cea0f75c3c3d3213539be5b22a7615ebe86532dcfaf5a4bf3f7
Nov 03 00:43:17.045527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:17.045495    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"]
Nov 03 00:43:17.296423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:17.295879    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:43:17.296423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:17.296042    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 00:43:17.296423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:17.296096    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 00:43:18.007174 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.007137    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" event={"ID":"658355d0-4279-4077-9d55-f38b8fb7d19b","Type":"ContainerStarted","Data":"4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693"}
Nov 03 00:43:18.007627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.007196    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" event={"ID":"658355d0-4279-4077-9d55-f38b8fb7d19b","Type":"ContainerStarted","Data":"26722ddd479c4cea0f75c3c3d3213539be5b22a7615ebe86532dcfaf5a4bf3f7"}
Nov 03 00:43:18.007627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.007236    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:18.047498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.047409    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" podStartSLOduration=2.047397366 podStartE2EDuration="2.047397366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:18.04687767 +0000 UTC m=+2280.368414472" watchObservedRunningTime="2026-11-03 00:43:18.047397366 +0000 UTC m=+2280.368934165"
Nov 03 00:43:18.288066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.287970    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 00:43:18.288654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.288619    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 00:43:18.339793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:18.339764    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:19.288521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:19.288475    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 00:43:19.288934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:19.288561    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 00:43:19.935520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:19.935490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:20.288498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:20.288398    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 00:43:20.288667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:20.288505    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 00:43:20.933501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:20.933469    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:43:21.288821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:21.288737    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Nov 03 00:43:21.289260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:21.289233    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Nov 03 00:43:21.603175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:21.603082    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:21.704963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:21.704927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:21.809564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:21.804824    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:22.311186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:22.311154    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:43:22.312462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:22.312376    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:43:22.312999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:22.312975    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 00:43:23.518924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:23.518885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:24.727518 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:43:24.727480    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:43:25.232340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:25.232292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:26.935825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:26.935788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:28.836373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:28.836339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:29.017043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:29.017015    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 00:43:30.485975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:30.485877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:31.675371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:31.675327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:32.168039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:32.168003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:33.112738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.112698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:33.543869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.543836    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"]
Nov 03 00:43:33.550801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.550777    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.587630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.587604    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-909268-openshift-default-dockercfg-8h6k7\""
Nov 03 00:43:33.595694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.595657    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"]
Nov 03 00:43:33.608637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.604819    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:33.684403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684372    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684411    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684479    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684561    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684690    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684717    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684732    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbt7\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684787    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.684915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.684819    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.705160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.705129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790258    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790324    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790349    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbt7\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790452    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790505    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790597    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790627    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790654    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.790702    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.791466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.791159    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.813895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.792354    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.813895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.792697    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.813895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.796944    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.813895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.799959    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.813895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.801229    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.813895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.806676    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.841464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.837750    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.856459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.848122    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbt7\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7\") pod \"e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") " pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:33.902782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:33.902730    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:34.035796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:34.035678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:34.340290 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:43:34.340251    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5e5569_e612_4dc2_b130_f3b8c8fb7b37.slice/crio-e9a89f887741966600aa75075710778d6155667355af735fb6b54e1248a34d60 WatchSource:0}: Error finding container e9a89f887741966600aa75075710778d6155667355af735fb6b54e1248a34d60: Status 404 returned error can't find the container with id e9a89f887741966600aa75075710778d6155667355af735fb6b54e1248a34d60
Nov 03 00:43:34.343540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:34.343495    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:34.343650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:34.343580    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:34.343650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:34.343609    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:34.346209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:34.346182    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"]
Nov 03 00:43:35.135748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:35.135708    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" event={"ID":"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37","Type":"ContainerStarted","Data":"275b9e73974cc46ff6e8e5972ae42fe7ee099bb7cac4551784e2a95f78073222"}
Nov 03 00:43:35.135953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:35.135778    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" event={"ID":"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37","Type":"ContainerStarted","Data":"e9a89f887741966600aa75075710778d6155667355af735fb6b54e1248a34d60"}
Nov 03 00:43:35.191155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:35.191109    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podStartSLOduration=2.191096256 podStartE2EDuration="2.191096256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:35.189841686 +0000 UTC m=+2297.511378484" watchObservedRunningTime="2026-11-03 00:43:35.191096256 +0000 UTC m=+2297.512633054"
Nov 03 00:43:35.641782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:35.641746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:35.904237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:35.904207    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:35.910478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:35.910426    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:36.142569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:36.142538    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:36.143426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:36.143409    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:43:37.519830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:37.519789    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:37.637342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:37.637314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:37.754579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:37.754545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:39.336829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:39.336788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:39.440190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:39.440157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:41.145914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:41.145873    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:42.800113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:42.800077    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:44.532882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:44.532850    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:46.207256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:46.207223    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:47.919981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:47.919942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:49.698556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:49.698522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:51.460854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:51.460820    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:53.119458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:53.116068    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:53.223985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:53.223948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:53.376606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:53.376503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:54.730668 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:43:54.730589    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:43:54.934546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:54.934506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:56.659124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:56.659088    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:57.953595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:57.948300    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": read tcp 10.133.0.2:56884->10.133.0.161:15021: read: connection reset by peer" start-of-body=
Nov 03 00:43:57.953595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:57.948357    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": read tcp 10.133.0.2:56884->10.133.0.161:15021: read: connection reset by peer"
Nov 03 00:43:58.347294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.347211    4550 generic.go:356] "Generic (PLEG): container finished" podID="91526e27-ec44-4306-aed5-67fb95bad247" containerID="e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05" exitCode=0
Nov 03 00:43:58.347294 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.347281    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" event={"ID":"91526e27-ec44-4306-aed5-67fb95bad247","Type":"ContainerDied","Data":"e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"}
Nov 03 00:43:58.347735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.347715    4550 scope.go:122] "RemoveContainer" containerID="e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"
Nov 03 00:43:58.348879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.348860    4550 generic.go:356] "Generic (PLEG): container finished" podID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerID="0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821" exitCode=0
Nov 03 00:43:58.348958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.348935    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" event={"ID":"7dff03b7-1b77-43f5-9204-29b75194a7d9","Type":"ContainerDied","Data":"0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"}
Nov 03 00:43:58.349277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.349263    4550 scope.go:122] "RemoveContainer" containerID="0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"
Nov 03 00:43:58.349571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.349548    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:58.349635 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.349622    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:58.349671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.349657    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:58.350930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.350904    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:58.350995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.350978    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:58.351035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.351017    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:43:58.517467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:58.516586    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:43:59.428326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:59.428284    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" event={"ID":"91526e27-ec44-4306-aed5-67fb95bad247","Type":"ContainerStarted","Data":"b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955"}
Nov 03 00:43:59.436698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:43:59.436667    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" event={"ID":"7dff03b7-1b77-43f5-9204-29b75194a7d9","Type":"ContainerStarted","Data":"b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a"}
Nov 03 00:44:00.129644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.129617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:00.228369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.225627    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:44:00.228369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.225761    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 00:44:00.228369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.225803    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 00:44:00.370843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.370814    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:44:00.371225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.371139    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 00:44:00.371408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:00.371384    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 00:44:01.067547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:01.067515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:01.221455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:01.220895    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 00:44:01.221455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:01.220975    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 00:44:01.376195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:01.376103    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 00:44:01.376466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:01.376421    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 00:44:01.888456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:01.887553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:02.222455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.221818    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 00:44:02.222455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.221874    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 00:44:02.325449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.325155    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:02.379453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.374080    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 00:44:02.379453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.374137    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 00:44:02.434206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.434168    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:02.608507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.608304    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:02.913456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:02.913288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:03.078328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.078288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:03.218540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.204109    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:03.220023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.219997    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 00:44:03.220238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.220215    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 00:44:03.371456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.370811    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 00:44:03.371456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.370882    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 00:44:03.698472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.694134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:03.815453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:03.814915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:04.224465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:04.221692    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body=
Nov 03 00:44:04.224465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:04.221753    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused"
Nov 03 00:44:04.375179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:04.375146    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 00:44:04.375793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:04.375763    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 00:44:04.620538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:04.620443    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:05.370990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:05.370962    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:44:05.371729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:05.371701    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:44:05.373693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:05.373669    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 00:44:05.375967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:05.375942    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:44:05.376357 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:05.376339    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:44:05.378070 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:05.378045    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 00:44:06.067141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:06.067094    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:07.475760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:07.475728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:07.578737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:07.578703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:08.044937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.044897    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"]
Nov 03 00:44:08.058787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.058761    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.092119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.092080    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"]
Nov 03 00:44:08.125059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.116347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:08.152173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.152147    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0db47e-openshift-default-dockercfg-7j9df\""
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.209479    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.209820    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.209903    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.209963    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.210004    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.210121    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.210164    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.210482    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.210893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.210593    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbdc\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.311679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311581    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.311679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311636    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.311918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311811    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.311918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311859    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbdc\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.311918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311914    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311949    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.311993    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312026    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312076    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312409    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312502    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312514    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312583    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.312678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.312630    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.314285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.314257    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.314415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.314401    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.331595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.331570    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbdc\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.331942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.331921    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token\") pod \"e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") " pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.338729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.338694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:08.399651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.399495    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:08.717173 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:44:08.717148    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05672915_3fd5_4fdd_af1b_36ad57cff170.slice/crio-c166e460339e171352a92efa3b879dd683c0dcddf62756ca861cc5e4b1327388 WatchSource:0}: Error finding container c166e460339e171352a92efa3b879dd683c0dcddf62756ca861cc5e4b1327388: Status 404 returned error can't find the container with id c166e460339e171352a92efa3b879dd683c0dcddf62756ca861cc5e4b1327388
Nov 03 00:44:08.718513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.718486    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"]
Nov 03 00:44:08.720928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.720891    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:08.721003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.720975    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:08.721041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:08.721017    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:09.325447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:09.325399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:09.650954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:09.650916    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" event={"ID":"05672915-3fd5-4fdd-af1b-36ad57cff170","Type":"ContainerStarted","Data":"1ec4bdb682700f9298dbb93ad66bab7c02add3faeb25332c8cd8cb927ff5338b"}
Nov 03 00:44:09.651258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:09.651238    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" event={"ID":"05672915-3fd5-4fdd-af1b-36ad57cff170","Type":"ContainerStarted","Data":"c166e460339e171352a92efa3b879dd683c0dcddf62756ca861cc5e4b1327388"}
Nov 03 00:44:09.721134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:09.717810    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podStartSLOduration=1.717779212 podStartE2EDuration="1.717779212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:09.716594387 +0000 UTC m=+2332.038131187" watchObservedRunningTime="2026-11-03 00:44:09.717779212 +0000 UTC m=+2332.039316012"
Nov 03 00:44:10.400402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:10.400378    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:10.405803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:10.405779    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:10.657979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:10.657893    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:10.658941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:10.658913    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:44:11.180453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:11.180153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:11.887795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:11.887760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:12.175150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:12.175110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:12.297531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:12.297497    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:13.124283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:13.124250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:13.818064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:13.817988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:13.968928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:13.968896    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:14.922656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:14.922618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:16.548216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:16.548174    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:18.204113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:18.204052    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:18.289807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:18.289769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:18.544485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:18.544385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:20.019665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.019619    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:20.749311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.749278    4550 generic.go:356] "Generic (PLEG): container finished" podID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerID="f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a" exitCode=0
Nov 03 00:44:20.749584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.749353    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" event={"ID":"48aeed2a-e1e5-4546-93bd-94db6c9953ae","Type":"ContainerDied","Data":"f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"}
Nov 03 00:44:20.749845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.749832    4550 scope.go:122] "RemoveContainer" containerID="f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"
Nov 03 00:44:20.751739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.751713    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:20.751830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.751783    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:20.751877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:20.751820    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:21.638995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:21.638963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:21.792953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:21.792912    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" event={"ID":"48aeed2a-e1e5-4546-93bd-94db6c9953ae","Type":"ContainerStarted","Data":"8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80"}
Nov 03 00:44:21.947703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:21.947665    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:44:21.948001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:21.947979    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 00:44:21.948068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:21.948028    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 00:44:22.948076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:22.948041    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 00:44:22.948526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:22.948096    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 00:44:23.347979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:23.347891    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:23.947563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:23.947531    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 00:44:23.947754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:23.947587    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 00:44:24.725291 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:44:24.725252    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:44:24.947851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:24.947814    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 00:44:24.948027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:24.947894    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 00:44:25.111632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:25.111529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:25.947689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:25.947655    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 00:44:25.948083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:25.947709    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 00:44:26.782131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:26.782090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:26.948026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:26.947988    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 03 00:44:26.948570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:26.948042    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 03 00:44:27.952421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:27.952390    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:44:27.952916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:27.952764    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:44:27.953461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:27.953444    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 00:44:28.376613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:28.376531    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:30.035920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:30.035886    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:31.723503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:31.723465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:33.351924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:33.351889    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:35.035066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:35.035022    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:36.829841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:36.829809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:38.544088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:38.544042    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:40.229107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:40.229074    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:41.448463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:41.448402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:42.165452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:42.164981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:42.833715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:42.833682    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:43.412853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.412813    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"]
Nov 03 00:44:43.420267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.420228    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.456014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.455983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:43.482761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.482711    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ac338d-openshift-default-dockercfg-hs9cf\""
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.550673    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.550757    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.550833    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.550861    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.550934    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.550979    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.551042    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298r2\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.551096    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.591327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.551169    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652554    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652620    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652694    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298r2\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652750    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652819    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652897    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.652965    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.653041    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.677203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.653068    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.693466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.688879    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.697494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.697462    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.698956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.698933    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.702681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.700237    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.733491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.729953    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.769512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.764041    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"]
Nov 03 00:44:43.769512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.769275    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.782121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.769765    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.815556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.815510    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:43.849447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:43.846117    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-298r2\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:44.013215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.013095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:44.060456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.060147    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:44.146775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.144179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:44.323445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.323337    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:44.455281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.455212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:44.503452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.501753    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"]
Nov 03 00:44:44.507556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.507511    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:44.507699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.507611    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:44.507699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.507645    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:44.554937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.554907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:44.752957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:44.752915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:45.022760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.022211    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" event={"ID":"96b3eb4a-d717-44b7-b0bc-18f8997e97dd","Type":"ContainerStarted","Data":"f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"}
Nov 03 00:44:45.022760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.022271    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" event={"ID":"96b3eb4a-d717-44b7-b0bc-18f8997e97dd","Type":"ContainerStarted","Data":"025e09d05db8b9ad5cc043a2e58f05f1a5f60d9e40953cc3c66e21bca76a01eb"}
Nov 03 00:44:45.064456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.062502    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:45.064456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.063336    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 00:44:45.064456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.063388    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 00:44:45.099971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.099910    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podStartSLOduration=2.099892547 podStartE2EDuration="2.099892547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:45.082144064 +0000 UTC m=+2367.403680863" watchObservedRunningTime="2026-11-03 00:44:45.099892547 +0000 UTC m=+2367.421429345"
Nov 03 00:44:45.394508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.393485    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:45.530453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:45.523268    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:46.074456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:46.073784    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:46.093579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:46.093547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:46.246454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:46.245963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:46.351509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:46.350631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:47.037086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:47.037041    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:47.038231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:47.038209    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:44:47.486455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:47.484963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:47.613164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:47.612988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:48.225326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:48.225287    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:49.193845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.193614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:49.312023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.311983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:49.313132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.313100    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"]
Nov 03 00:44:49.318682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.318661    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.335993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.335955    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"]
Nov 03 00:44:49.343547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.343480    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ac338d\""
Nov 03 00:44:49.343730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.343682    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ac338d-dockercfg-j74f2\""
Nov 03 00:44:49.438454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.422411    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.438454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.422641    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zt9z\" (UniqueName: \"kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z\") pod \"payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.552520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.535739    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.552520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.536028    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zt9z\" (UniqueName: \"kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z\") pod \"payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.552520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.544759    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.585471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.583621    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zt9z\" (UniqueName: \"kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z\") pod \"payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.639275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.639237    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:49.965259 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:44:49.965218    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44416698_1c57_40cc_9094_ed0e937081a7.slice/crio-fdfc2d58572a15f1bc70db45e752181d50edb1be1f7c72aa307ddfb4f4e7eddd WatchSource:0}: Error finding container fdfc2d58572a15f1bc70db45e752181d50edb1be1f7c72aa307ddfb4f4e7eddd: Status 404 returned error can't find the container with id fdfc2d58572a15f1bc70db45e752181d50edb1be1f7c72aa307ddfb4f4e7eddd
Nov 03 00:44:49.966984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:49.966956    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"]
Nov 03 00:44:50.061893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:50.061859    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn" event={"ID":"44416698-1c57-40cc-9094-ed0e937081a7","Type":"ContainerStarted","Data":"d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f"}
Nov 03 00:44:50.062097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:50.061924    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn" event={"ID":"44416698-1c57-40cc-9094-ed0e937081a7","Type":"ContainerStarted","Data":"fdfc2d58572a15f1bc70db45e752181d50edb1be1f7c72aa307ddfb4f4e7eddd"}
Nov 03 00:44:50.062382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:50.062266    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:44:50.105060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:50.105000    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn" podStartSLOduration=1.104986019 podStartE2EDuration="1.104986019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:50.103622587 +0000 UTC m=+2372.425159386" watchObservedRunningTime="2026-11-03 00:44:50.104986019 +0000 UTC m=+2372.426522833"
Nov 03 00:44:50.149551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:50.149515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:51.611913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:51.610652    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": read tcp 10.133.0.2:39018->10.133.0.169:15021: read: connection reset by peer" start-of-body=
Nov 03 00:44:51.611913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:51.610740    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": read tcp 10.133.0.2:39018->10.133.0.169:15021: read: connection reset by peer"
Nov 03 00:44:51.808337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:51.808233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:52.094279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:52.094229    4550 generic.go:356] "Generic (PLEG): container finished" podID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerID="275b9e73974cc46ff6e8e5972ae42fe7ee099bb7cac4551784e2a95f78073222" exitCode=0
Nov 03 00:44:52.094519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:52.094313    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" event={"ID":"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37","Type":"ContainerDied","Data":"275b9e73974cc46ff6e8e5972ae42fe7ee099bb7cac4551784e2a95f78073222"}
Nov 03 00:44:52.094781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:52.094765    4550 scope.go:122] "RemoveContainer" containerID="275b9e73974cc46ff6e8e5972ae42fe7ee099bb7cac4551784e2a95f78073222"
Nov 03 00:44:52.096536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:52.096507    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:52.096665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:52.096574    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:52.096665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:52.096603    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:44:53.104210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.104172    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" event={"ID":"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37","Type":"ContainerStarted","Data":"3b39f6071d80aa5abdc3b8aa28c54f85b700993b268ec29828c115fd003c455b"}
Nov 03 00:44:53.591456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.590486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:53.691452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.690033    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:53.804495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.799278    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:53.904600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.904570    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:44:53.904845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.904816    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 00:44:53.904915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:53.904869    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 00:44:54.875456 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:44:54.866182    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:44:54.905178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:54.905111    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 00:44:54.905354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:54.905196    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 00:44:55.390225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:55.390125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:55.904500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:55.904461    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 00:44:55.904902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:55.904526    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 00:44:56.905065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:56.905030    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 00:44:56.905483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:56.905093    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 00:44:57.248590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:57.248492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:57.905273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:57.905237    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Nov 03 00:44:57.905740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:57.905289    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Nov 03 00:44:58.909483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:58.909427    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:44:58.909869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:58.909837    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:44:58.910755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:58.910732    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 00:44:58.928088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:58.928060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:44:59.834582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:44:59.834351    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"]
Nov 03 00:45:00.171901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.171866    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"]
Nov 03 00:45:00.177214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.177184    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"]
Nov 03 00:45:00.177410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.177307    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.197368 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.197338    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"]
Nov 03 00:45:00.197661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.197642    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.204024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.203983    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"]
Nov 03 00:45:00.204782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.204719    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.208456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.208416    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:45:00.230602 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.230568    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"]
Nov 03 00:45:00.230863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.230843    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.246656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.246625    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"]
Nov 03 00:45:00.246930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.246732    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.252032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.252003    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"]
Nov 03 00:45:00.252200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.252046    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"]
Nov 03 00:45:00.252200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.252120    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.255974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.255863    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"]
Nov 03 00:45:00.257748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.257715    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.257977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.257940    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.259164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.259141    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.259895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.259856    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.260057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.260039    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4n8l\" (UniqueName: \"kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.261133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.261093    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gvzt\" (UniqueName: \"kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.261456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.261426    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hhm\" (UniqueName: \"kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.261677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.261662    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhsv\" (UniqueName: \"kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.267126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.266519    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"]
Nov 03 00:45:00.271082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.270616    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.271082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.270770    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"]
Nov 03 00:45:00.277685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.277620    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.287176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.284756    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"]
Nov 03 00:45:00.287176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.285751    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"]
Nov 03 00:45:00.292741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.291847    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"]
Nov 03 00:45:00.303692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.297062    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.303692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.297197    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"]
Nov 03 00:45:00.311309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.310665    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"]
Nov 03 00:45:00.313799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.313445    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.319883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.319855    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"]
Nov 03 00:45:00.320050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.319911    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"]
Nov 03 00:45:00.320050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.320005    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.332740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.332708    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"]
Nov 03 00:45:00.345485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.340831    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.345485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.341526    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"]
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.352248    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"]
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.352383    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363686    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363732    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4fb\" (UniqueName: \"kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363784    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363829    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363858    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363901    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363938    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.363981    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364013    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364066    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n8l\" (UniqueName: \"kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364118    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364163    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gvzt\" (UniqueName: \"kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364194    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frm47\" (UniqueName: \"kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364249    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kbp\" (UniqueName: \"kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364330    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.367853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.364378    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hhm\" (UniqueName: \"kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.370606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.370536    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"]
Nov 03 00:45:00.381705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.380238    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.382443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.382309    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.386758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.385711    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.386758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.386363    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.418480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.386956    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhsv\" (UniqueName: \"kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.418480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.386998    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m89t\" (UniqueName: \"kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.418480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.387082    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxgm\" (UniqueName: \"kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.418480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.387115    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g2nw\" (UniqueName: \"kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.418480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.387184    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bvx\" (UniqueName: \"kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.424331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.423171    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4n8l\" (UniqueName: \"kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.424331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.423171    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gvzt\" (UniqueName: \"kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.424331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.423914    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hhm\" (UniqueName: \"kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm\") pod \"maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.427231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.426201    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhsv\" (UniqueName: \"kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.427231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.426247    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"]
Nov 03 00:45:00.432571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.432539    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"]
Nov 03 00:45:00.445248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.443633    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"]
Nov 03 00:45:00.454925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.449475    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"]
Nov 03 00:45:00.454925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.449585    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.454925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.453503    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"]
Nov 03 00:45:00.465387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.465311    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"]
Nov 03 00:45:00.475107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.470868    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"]
Nov 03 00:45:00.475107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.471005    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.475107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.472666    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"]
Nov 03 00:45:00.480347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.480327    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.488524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.488487    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m89t\" (UniqueName: \"kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.488762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.488743    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxgm\" (UniqueName: \"kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.488885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.488864    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.489004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.488980    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g2nw\" (UniqueName: \"kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.489138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.489029    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.490129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490102    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bvx\" (UniqueName: \"kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.490248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490150    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.490248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490212    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.490248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490242    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4fb\" (UniqueName: \"kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.490463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490301    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.490463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490352    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.490463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490396    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.490639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490478    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.490639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490541    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.490639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490593    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frm47\" (UniqueName: \"kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.490791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490655    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46kbp\" (UniqueName: \"kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.490791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490727    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkzr\" (UniqueName: \"kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.490791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.490761    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.491636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.491610    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.492292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.491955    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.492896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.492877    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"]
Nov 03 00:45:00.493455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.493415    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.494489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.493672    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.494489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.494075    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.494489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.494089    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.494489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.494247    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.496035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.496012    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"]
Nov 03 00:45:00.513224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.513192    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frm47\" (UniqueName: \"kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.515410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.515387    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bvx\" (UniqueName: \"kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.516166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.516144    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4fb\" (UniqueName: \"kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.516676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.516657    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m89t\" (UniqueName: \"kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.516829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.516809    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g2nw\" (UniqueName: \"kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.516891 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.516871    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxgm\" (UniqueName: \"kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.518841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.518812    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kbp\" (UniqueName: \"kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.527730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.527706    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:00.540561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.540526    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:00.557717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.557689    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:00.600156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.600122    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.610830    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.610979    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611012    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2szn\" (UniqueName: \"kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611046    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611262    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx4r\" (UniqueName: \"kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611337    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkzr\" (UniqueName: \"kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611377    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611572    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611642    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.611681    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wzlq\" (UniqueName: \"kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.612629    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.616049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.614318    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.621799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.621776    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:00.645887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.645602    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:00.653809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.653781    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkzr\" (UniqueName: \"kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.670673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.670604    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.680125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.679888    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:00.728497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.727407    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.728497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.727494    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2szn\" (UniqueName: \"kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.728497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.727525    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.728497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.727641    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx4r\" (UniqueName: \"kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.728497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.727735    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.728497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.727919    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wzlq\" (UniqueName: \"kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.731761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.729546    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.731761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.730501    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.733299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.732147    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.733299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.733025    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:00.743820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.735221    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:00.766959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.766626    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2szn\" (UniqueName: \"kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:00.768423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.768368    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wzlq\" (UniqueName: \"kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.779912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.779509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx4r\" (UniqueName: \"kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:00.779912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.779754    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:00.796052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.795795    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:00.820033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.811993    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:00.855724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.852484    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:00.925127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.924758    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:00.934332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.933286    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:00.953023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:00.952785    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:01.009347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.008705    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:01.090076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.090003    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"]
Nov 03 00:45:01.105478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.102950    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 00:45:01.183250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.183217    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerStarted","Data":"1525b5d2b8170837b98cce23ef5c2d46738a8ce1daec3ea5daadf1d46211ad92"}
Nov 03 00:45:01.332733 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:01.332690    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8abcb6_5936_438c_83a5_7050131bb1c5.slice/crio-efe091c9f35bae8052a0108b46ac174e9a482d207116c290ff48cc76be125741 WatchSource:0}: Error finding container efe091c9f35bae8052a0108b46ac174e9a482d207116c290ff48cc76be125741: Status 404 returned error can't find the container with id efe091c9f35bae8052a0108b46ac174e9a482d207116c290ff48cc76be125741
Nov 03 00:45:01.336536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.336515    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"]
Nov 03 00:45:01.414731 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:01.414697    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c9eb1c_745a_403f_ae58_29e5b6c6d4db.slice/crio-ac5216cf3537b9088c3ad7a80bb4213a28130bb626109df84a50d38ffe50d6b7 WatchSource:0}: Error finding container ac5216cf3537b9088c3ad7a80bb4213a28130bb626109df84a50d38ffe50d6b7: Status 404 returned error can't find the container with id ac5216cf3537b9088c3ad7a80bb4213a28130bb626109df84a50d38ffe50d6b7
Nov 03 00:45:01.420146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.418230    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"]
Nov 03 00:45:01.551056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.551028    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"]
Nov 03 00:45:01.558213 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:01.558184    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ae6ba2_9455_4a12_a5ca_b5064d128f34.slice/crio-ba0ca98f9846cbba2511a5bce7d6adf13ed1708e3f08d2ae299ac927e92f0f07 WatchSource:0}: Error finding container ba0ca98f9846cbba2511a5bce7d6adf13ed1708e3f08d2ae299ac927e92f0f07: Status 404 returned error can't find the container with id ba0ca98f9846cbba2511a5bce7d6adf13ed1708e3f08d2ae299ac927e92f0f07
Nov 03 00:45:01.567553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.567527    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"]
Nov 03 00:45:01.573557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.573271    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"]
Nov 03 00:45:01.581366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.581338    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"]
Nov 03 00:45:01.586074 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:01.586049    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb758cbeb_2873_4a60_8781_ec74cd985a5f.slice/crio-c2065faf24e7df700e17e5b2ac15584187d0c4a15a10e624defc59d4e9f8af66 WatchSource:0}: Error finding container c2065faf24e7df700e17e5b2ac15584187d0c4a15a10e624defc59d4e9f8af66: Status 404 returned error can't find the container with id c2065faf24e7df700e17e5b2ac15584187d0c4a15a10e624defc59d4e9f8af66
Nov 03 00:45:01.614035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.613999    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"]
Nov 03 00:45:01.622811 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:01.622771    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9a0a98_a505_4165_be89_7b45f49226e8.slice/crio-84a60ada027f66af4f0f15c1274a676aa05dca293ae59496beda7b1e47ced2bc WatchSource:0}: Error finding container 84a60ada027f66af4f0f15c1274a676aa05dca293ae59496beda7b1e47ced2bc: Status 404 returned error can't find the container with id 84a60ada027f66af4f0f15c1274a676aa05dca293ae59496beda7b1e47ced2bc
Nov 03 00:45:01.630787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.623137    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"]
Nov 03 00:45:01.640751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.640725    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"]
Nov 03 00:45:01.664396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.664364    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"]
Nov 03 00:45:01.693898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.693867    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"]
Nov 03 00:45:01.704797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.704772    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"]
Nov 03 00:45:01.708209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.708185    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"]
Nov 03 00:45:01.716539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.716514    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"]
Nov 03 00:45:01.732223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:01.732191    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"]
Nov 03 00:45:01.761008 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:01.760964    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9634e52_dcc3_4e14_b7ee_bfe7f207c870.slice/crio-90a218e90c50e8f5346aae90fad8c7ce653387d1c90c32973ce8c0791609a32f WatchSource:0}: Error finding container 90a218e90c50e8f5346aae90fad8c7ce653387d1c90c32973ce8c0791609a32f: Status 404 returned error can't find the container with id 90a218e90c50e8f5346aae90fad8c7ce653387d1c90c32973ce8c0791609a32f
Nov 03 00:45:02.159205 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:02.159171    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8014b63_70fd_4992_aa3d_7a6664cea4bd.slice/crio-ef539f4def2ab876e5d787039f6d79ff0bc860aa06e243a7c116deaaa03858d4 WatchSource:0}: Error finding container ef539f4def2ab876e5d787039f6d79ff0bc860aa06e243a7c116deaaa03858d4: Status 404 returned error can't find the container with id ef539f4def2ab876e5d787039f6d79ff0bc860aa06e243a7c116deaaa03858d4
Nov 03 00:45:02.162409 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:02.162383    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044c984_2256_46c0_a939_bb5d39f2a9f0.slice/crio-8b74ffd8b295cf719f924a12f97b560cdba3b72149c4faf998547a30fe58c74c WatchSource:0}: Error finding container 8b74ffd8b295cf719f924a12f97b560cdba3b72149c4faf998547a30fe58c74c: Status 404 returned error can't find the container with id 8b74ffd8b295cf719f924a12f97b560cdba3b72149c4faf998547a30fe58c74c
Nov 03 00:45:02.196864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.196831    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerStarted","Data":"b3ea31d60790030e49764e2d0ae8da98371927a34aee5a8a0a417324eff3f4d2"}
Nov 03 00:45:02.198262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.198236    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerStarted","Data":"90a218e90c50e8f5346aae90fad8c7ce653387d1c90c32973ce8c0791609a32f"}
Nov 03 00:45:02.199563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.199534    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerStarted","Data":"c2065faf24e7df700e17e5b2ac15584187d0c4a15a10e624defc59d4e9f8af66"}
Nov 03 00:45:02.200754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.200733    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerStarted","Data":"ef539f4def2ab876e5d787039f6d79ff0bc860aa06e243a7c116deaaa03858d4"}
Nov 03 00:45:02.201876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.201854    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerStarted","Data":"8b74ffd8b295cf719f924a12f97b560cdba3b72149c4faf998547a30fe58c74c"}
Nov 03 00:45:02.202987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.202967    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerStarted","Data":"5bcfd8478b41dfefbac6c92bab286348a559a21d478f254573fd8036e3b99ec7"}
Nov 03 00:45:02.204116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.204096    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerStarted","Data":"84a60ada027f66af4f0f15c1274a676aa05dca293ae59496beda7b1e47ced2bc"}
Nov 03 00:45:02.205215 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.205195    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerStarted","Data":"0ebcd0d7c6f1ee8d5051421c8cb4bb4aadd11ec206b862fbbb296eeb9048c28a"}
Nov 03 00:45:02.206315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.206296    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" event={"ID":"2cc1e19a-e169-4974-a452-3a9446f4b3ac","Type":"ContainerStarted","Data":"327902daf65befedbaa8aa45db71095da127bf22713e271f8bb033d4d903b507"}
Nov 03 00:45:02.207462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.207409    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerStarted","Data":"ba0ca98f9846cbba2511a5bce7d6adf13ed1708e3f08d2ae299ac927e92f0f07"}
Nov 03 00:45:02.208851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.208828    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerStarted","Data":"13987fa7bd9986169038ad1f669215d3cb9f52dd43126d678662d8854266e4d8"}
Nov 03 00:45:02.208935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.208867    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerStarted","Data":"efe091c9f35bae8052a0108b46ac174e9a482d207116c290ff48cc76be125741"}
Nov 03 00:45:02.210257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.210238    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerStarted","Data":"f78c3246e4ad9935f203240e2d114ac699d1a7f228c21c27e6ebe84535e237a6"}
Nov 03 00:45:02.211534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.211516    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerStarted","Data":"951ec701ee52b061ea25e82e7149faca788ab895e98e3ef38170e98c37106a27"}
Nov 03 00:45:02.212719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.212700    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" event={"ID":"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db","Type":"ContainerStarted","Data":"ac5216cf3537b9088c3ad7a80bb4213a28130bb626109df84a50d38ffe50d6b7"}
Nov 03 00:45:02.260527 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:02.260405    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7524e5_60fd_4d8a_8cac_75bd14006e93.slice/crio-f58db1aec9a1f74f9c8af7ee9c228bba8a04e549d8cac79c445bf327fcb06ab5 WatchSource:0}: Error finding container f58db1aec9a1f74f9c8af7ee9c228bba8a04e549d8cac79c445bf327fcb06ab5: Status 404 returned error can't find the container with id f58db1aec9a1f74f9c8af7ee9c228bba8a04e549d8cac79c445bf327fcb06ab5
Nov 03 00:45:02.261586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.261527    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" podStartSLOduration=2.261509148 podStartE2EDuration="2.261509148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:02.253881781 +0000 UTC m=+2384.575418581" watchObservedRunningTime="2026-11-03 00:45:02.261509148 +0000 UTC m=+2384.583045948"
Nov 03 00:45:02.265951 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:02.265919    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a596167_f57e_4c78_8d4f_d817b460d090.slice/crio-99eb072228cc770287e03ac8d8099a07270b1492fe169543716d91306b53260a WatchSource:0}: Error finding container 99eb072228cc770287e03ac8d8099a07270b1492fe169543716d91306b53260a: Status 404 returned error can't find the container with id 99eb072228cc770287e03ac8d8099a07270b1492fe169543716d91306b53260a
Nov 03 00:45:02.299250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.299201    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" podStartSLOduration=2.299184686 podStartE2EDuration="2.299184686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:02.2955047 +0000 UTC m=+2384.617041523" watchObservedRunningTime="2026-11-03 00:45:02.299184686 +0000 UTC m=+2384.620721489"
Nov 03 00:45:02.644446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:02.644399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:03.229784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.226107    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerStarted","Data":"8662e9619bdfbfd9f72270674e7e94b720a4b219afb61063f92ff8f96419dcd2"}
Nov 03 00:45:03.229784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.228266    4550 generic.go:356] "Generic (PLEG): container finished" podID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerID="13987fa7bd9986169038ad1f669215d3cb9f52dd43126d678662d8854266e4d8" exitCode=60
Nov 03 00:45:03.229784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.228347    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerDied","Data":"13987fa7bd9986169038ad1f669215d3cb9f52dd43126d678662d8854266e4d8"}
Nov 03 00:45:03.229784 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.228606    4550 scope.go:122] "RemoveContainer" containerID="13987fa7bd9986169038ad1f669215d3cb9f52dd43126d678662d8854266e4d8"
Nov 03 00:45:03.230688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.230667    4550 generic.go:356] "Generic (PLEG): container finished" podID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerID="f78c3246e4ad9935f203240e2d114ac699d1a7f228c21c27e6ebe84535e237a6" exitCode=60
Nov 03 00:45:03.230773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.230732    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerDied","Data":"f78c3246e4ad9935f203240e2d114ac699d1a7f228c21c27e6ebe84535e237a6"}
Nov 03 00:45:03.230998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.230975    4550 scope.go:122] "RemoveContainer" containerID="f78c3246e4ad9935f203240e2d114ac699d1a7f228c21c27e6ebe84535e237a6"
Nov 03 00:45:03.232572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.232550    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerStarted","Data":"7deeb323b8bc08bed811c40925e3fad0c34be868463bd1dc503c3f8269cd3028"}
Nov 03 00:45:03.234238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.234215    4550 generic.go:356] "Generic (PLEG): container finished" podID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerID="0c5fc93363728e0f770805b2617ad73de578b132fa686e124cfb5481bf2c5048" exitCode=60
Nov 03 00:45:03.234387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.234289    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" event={"ID":"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db","Type":"ContainerDied","Data":"0c5fc93363728e0f770805b2617ad73de578b132fa686e124cfb5481bf2c5048"}
Nov 03 00:45:03.234512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.234498    4550 scope.go:122] "RemoveContainer" containerID="0c5fc93363728e0f770805b2617ad73de578b132fa686e124cfb5481bf2c5048"
Nov 03 00:45:03.235881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.235856    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerStarted","Data":"fe7cfd7b6acee1160aa02cfa93259ab4536236ea4fda03805598df07fa0ee6d7"}
Nov 03 00:45:03.237356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.237333    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerStarted","Data":"e4139d152a5ca5c3705ba3d8266073a038fdff39dc580efc36eb8a624238d8c7"}
Nov 03 00:45:03.238841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.238823    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerStarted","Data":"5609ac200709eb42bb0d803f920658981b0ee61c3186b28e964730b55f611612"}
Nov 03 00:45:03.240376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.240353    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerStarted","Data":"f7154f0c55b1c417b93d053a6927c9bcacc53b5cf777f59006d53223f7a12080"}
Nov 03 00:45:03.241652 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.241633    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerStarted","Data":"f58db1aec9a1f74f9c8af7ee9c228bba8a04e549d8cac79c445bf327fcb06ab5"}
Nov 03 00:45:03.243135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.243110    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerStarted","Data":"962810a61c4c350acb50bc36097a8627439f9c2f4ab3bb07df8b10365adcd825"}
Nov 03 00:45:03.243273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.243150    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerStarted","Data":"99eb072228cc770287e03ac8d8099a07270b1492fe169543716d91306b53260a"}
Nov 03 00:45:03.244560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.244534    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerStarted","Data":"4f728c72168d2931935f2d972f9a20d04cd6a9d3b75d39c178c14048edc9ab92"}
Nov 03 00:45:03.246722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.246700    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerStarted","Data":"47d30963176408d7ebed5ea64de121a26a1a1f4c8695b9fb7744e09996d4263c"}
Nov 03 00:45:03.248173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.248140    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerStarted","Data":"54ebb7ded60cf7d76b09ccbeda6402d00a688f895337d83d62197aa8575738c6"}
Nov 03 00:45:03.249594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.249572    4550 generic.go:356] "Generic (PLEG): container finished" podID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerID="9e272bac97a350133abe518ab05b22897d46d492cd4d1dc9d11ae6bceb2fd251" exitCode=60
Nov 03 00:45:03.249681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.249603    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" event={"ID":"2cc1e19a-e169-4974-a452-3a9446f4b3ac","Type":"ContainerDied","Data":"9e272bac97a350133abe518ab05b22897d46d492cd4d1dc9d11ae6bceb2fd251"}
Nov 03 00:45:03.249794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.249779    4550 scope.go:122] "RemoveContainer" containerID="9e272bac97a350133abe518ab05b22897d46d492cd4d1dc9d11ae6bceb2fd251"
Nov 03 00:45:03.270644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.270390    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" podStartSLOduration=3.270371451 podStartE2EDuration="3.270371451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.26802556 +0000 UTC m=+2385.589562360" watchObservedRunningTime="2026-11-03 00:45:03.270371451 +0000 UTC m=+2385.591908251"
Nov 03 00:45:03.312522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.312467    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" podStartSLOduration=3.312451164 podStartE2EDuration="3.312451164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.308324838 +0000 UTC m=+2385.629861631" watchObservedRunningTime="2026-11-03 00:45:03.312451164 +0000 UTC m=+2385.633987959"
Nov 03 00:45:03.349587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.349536    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" podStartSLOduration=3.349523108 podStartE2EDuration="3.349523108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.347145359 +0000 UTC m=+2385.668682177" watchObservedRunningTime="2026-11-03 00:45:03.349523108 +0000 UTC m=+2385.671059910"
Nov 03 00:45:03.430177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.430119    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" podStartSLOduration=3.430105776 podStartE2EDuration="3.430105776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.428117012 +0000 UTC m=+2385.749653832" watchObservedRunningTime="2026-11-03 00:45:03.430105776 +0000 UTC m=+2385.751642612"
Nov 03 00:45:03.605783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.605734    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" podStartSLOduration=3.605720423 podStartE2EDuration="3.605720423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.60148427 +0000 UTC m=+2385.923021090" watchObservedRunningTime="2026-11-03 00:45:03.605720423 +0000 UTC m=+2385.927257220"
Nov 03 00:45:03.667226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.666992    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" podStartSLOduration=3.6669735919999997 podStartE2EDuration="3.666973592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.659365738 +0000 UTC m=+2385.980902559" watchObservedRunningTime="2026-11-03 00:45:03.666973592 +0000 UTC m=+2385.988510456"
Nov 03 00:45:03.716167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.716100    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" podStartSLOduration=3.716082557 podStartE2EDuration="3.716082557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.714476479 +0000 UTC m=+2386.036013276" watchObservedRunningTime="2026-11-03 00:45:03.716082557 +0000 UTC m=+2386.037619357"
Nov 03 00:45:03.755484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.755405    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" podStartSLOduration=3.755391698 podStartE2EDuration="3.755391698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.753558001 +0000 UTC m=+2386.075094827" watchObservedRunningTime="2026-11-03 00:45:03.755391698 +0000 UTC m=+2386.076928538"
Nov 03 00:45:03.837845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.837793    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" podStartSLOduration=3.8377779 podStartE2EDuration="3.8377779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.834010655 +0000 UTC m=+2386.155547453" watchObservedRunningTime="2026-11-03 00:45:03.8377779 +0000 UTC m=+2386.159314698"
Nov 03 00:45:03.877987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:03.877923    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" podStartSLOduration=3.8779026 podStartE2EDuration="3.8779026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:03.873253855 +0000 UTC m=+2386.194790678" watchObservedRunningTime="2026-11-03 00:45:03.8779026 +0000 UTC m=+2386.199439398"
Nov 03 00:45:04.354413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.354370    4550 generic.go:356] "Generic (PLEG): container finished" podID="2a596167-f57e-4c78-8d4f-d817b460d090" containerID="962810a61c4c350acb50bc36097a8627439f9c2f4ab3bb07df8b10365adcd825" exitCode=60
Nov 03 00:45:04.354850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.354645    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerDied","Data":"962810a61c4c350acb50bc36097a8627439f9c2f4ab3bb07df8b10365adcd825"}
Nov 03 00:45:04.355246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.355123    4550 scope.go:122] "RemoveContainer" containerID="962810a61c4c350acb50bc36097a8627439f9c2f4ab3bb07df8b10365adcd825"
Nov 03 00:45:04.362738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.362709    4550 generic.go:356] "Generic (PLEG): container finished" podID="56666705-dfb5-4876-b27b-22ee10e32c11" containerID="4f728c72168d2931935f2d972f9a20d04cd6a9d3b75d39c178c14048edc9ab92" exitCode=60
Nov 03 00:45:04.362845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.362816    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerDied","Data":"4f728c72168d2931935f2d972f9a20d04cd6a9d3b75d39c178c14048edc9ab92"}
Nov 03 00:45:04.363333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.363312    4550 scope.go:122] "RemoveContainer" containerID="4f728c72168d2931935f2d972f9a20d04cd6a9d3b75d39c178c14048edc9ab92"
Nov 03 00:45:04.366930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.366904    4550 generic.go:356] "Generic (PLEG): container finished" podID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerID="47d30963176408d7ebed5ea64de121a26a1a1f4c8695b9fb7744e09996d4263c" exitCode=60
Nov 03 00:45:04.370323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.367150    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerDied","Data":"47d30963176408d7ebed5ea64de121a26a1a1f4c8695b9fb7744e09996d4263c"}
Nov 03 00:45:04.370323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.368070    4550 scope.go:122] "RemoveContainer" containerID="47d30963176408d7ebed5ea64de121a26a1a1f4c8695b9fb7744e09996d4263c"
Nov 03 00:45:04.373143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.373113    4550 generic.go:356] "Generic (PLEG): container finished" podID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerID="54ebb7ded60cf7d76b09ccbeda6402d00a688f895337d83d62197aa8575738c6" exitCode=60
Nov 03 00:45:04.373695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.373347    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerDied","Data":"54ebb7ded60cf7d76b09ccbeda6402d00a688f895337d83d62197aa8575738c6"}
Nov 03 00:45:04.373695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.373565    4550 scope.go:122] "RemoveContainer" containerID="54ebb7ded60cf7d76b09ccbeda6402d00a688f895337d83d62197aa8575738c6"
Nov 03 00:45:04.377968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.377935    4550 generic.go:356] "Generic (PLEG): container finished" podID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerID="8662e9619bdfbfd9f72270674e7e94b720a4b219afb61063f92ff8f96419dcd2" exitCode=60
Nov 03 00:45:04.378040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.377985    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerDied","Data":"8662e9619bdfbfd9f72270674e7e94b720a4b219afb61063f92ff8f96419dcd2"}
Nov 03 00:45:04.378482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.378251    4550 scope.go:122] "RemoveContainer" containerID="8662e9619bdfbfd9f72270674e7e94b720a4b219afb61063f92ff8f96419dcd2"
Nov 03 00:45:04.386086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.386058    4550 generic.go:356] "Generic (PLEG): container finished" podID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerID="7deeb323b8bc08bed811c40925e3fad0c34be868463bd1dc503c3f8269cd3028" exitCode=60
Nov 03 00:45:04.386244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.386121    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerDied","Data":"7deeb323b8bc08bed811c40925e3fad0c34be868463bd1dc503c3f8269cd3028"}
Nov 03 00:45:04.386741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.386555    4550 scope.go:122] "RemoveContainer" containerID="7deeb323b8bc08bed811c40925e3fad0c34be868463bd1dc503c3f8269cd3028"
Nov 03 00:45:04.398581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.398555    4550 generic.go:356] "Generic (PLEG): container finished" podID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerID="fe7cfd7b6acee1160aa02cfa93259ab4536236ea4fda03805598df07fa0ee6d7" exitCode=60
Nov 03 00:45:04.398759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.398744    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerDied","Data":"fe7cfd7b6acee1160aa02cfa93259ab4536236ea4fda03805598df07fa0ee6d7"}
Nov 03 00:45:04.399251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.399228    4550 scope.go:122] "RemoveContainer" containerID="fe7cfd7b6acee1160aa02cfa93259ab4536236ea4fda03805598df07fa0ee6d7"
Nov 03 00:45:04.421946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.421922    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerID="e4139d152a5ca5c3705ba3d8266073a038fdff39dc580efc36eb8a624238d8c7" exitCode=60
Nov 03 00:45:04.422178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.422160    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerDied","Data":"e4139d152a5ca5c3705ba3d8266073a038fdff39dc580efc36eb8a624238d8c7"}
Nov 03 00:45:04.422656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.422640    4550 scope.go:122] "RemoveContainer" containerID="e4139d152a5ca5c3705ba3d8266073a038fdff39dc580efc36eb8a624238d8c7"
Nov 03 00:45:04.436393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.436366    4550 generic.go:356] "Generic (PLEG): container finished" podID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerID="5609ac200709eb42bb0d803f920658981b0ee61c3186b28e964730b55f611612" exitCode=60
Nov 03 00:45:04.436609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.436585    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerDied","Data":"5609ac200709eb42bb0d803f920658981b0ee61c3186b28e964730b55f611612"}
Nov 03 00:45:04.437491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.436953    4550 scope.go:122] "RemoveContainer" containerID="5609ac200709eb42bb0d803f920658981b0ee61c3186b28e964730b55f611612"
Nov 03 00:45:04.454060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.451675    4550 generic.go:356] "Generic (PLEG): container finished" podID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerID="f7154f0c55b1c417b93d053a6927c9bcacc53b5cf777f59006d53223f7a12080" exitCode=60
Nov 03 00:45:04.454060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.451773    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerDied","Data":"f7154f0c55b1c417b93d053a6927c9bcacc53b5cf777f59006d53223f7a12080"}
Nov 03 00:45:04.454060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.452212    4550 scope.go:122] "RemoveContainer" containerID="f7154f0c55b1c417b93d053a6927c9bcacc53b5cf777f59006d53223f7a12080"
Nov 03 00:45:04.459148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.459120    4550 generic.go:356] "Generic (PLEG): container finished" podID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerID="35a98163d3843b12f0b80886e06d9d4f026c8889d717ba856767d375ef74d021" exitCode=60
Nov 03 00:45:04.459344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.459326    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerDied","Data":"35a98163d3843b12f0b80886e06d9d4f026c8889d717ba856767d375ef74d021"}
Nov 03 00:45:04.459867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.459848    4550 scope.go:122] "RemoveContainer" containerID="35a98163d3843b12f0b80886e06d9d4f026c8889d717ba856767d375ef74d021"
Nov 03 00:45:04.469970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.469942    4550 generic.go:356] "Generic (PLEG): container finished" podID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerID="ced70645927abfa36bd5d39f6c09655bf6c950413bc317ce12eddd2055df4c10" exitCode=60
Nov 03 00:45:04.470114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.470098    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerDied","Data":"ced70645927abfa36bd5d39f6c09655bf6c950413bc317ce12eddd2055df4c10"}
Nov 03 00:45:04.470421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.470400    4550 scope.go:122] "RemoveContainer" containerID="ced70645927abfa36bd5d39f6c09655bf6c950413bc317ce12eddd2055df4c10"
Nov 03 00:45:04.583768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.582877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:04.642770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.640708    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" podStartSLOduration=4.640692359 podStartE2EDuration="4.640692359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:04.639608513 +0000 UTC m=+2386.961145313" watchObservedRunningTime="2026-11-03 00:45:04.640692359 +0000 UTC m=+2386.962229162"
Nov 03 00:45:04.698486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.698078    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" podStartSLOduration=4.698065004 podStartE2EDuration="4.698065004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:04.697103701 +0000 UTC m=+2387.018640500" watchObservedRunningTime="2026-11-03 00:45:04.698065004 +0000 UTC m=+2387.019601866"
Nov 03 00:45:04.834978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:04.834890    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7" podUID="18de8fc3-d694-4f27-935d-bef519be6005" containerName="payload-processing" containerID="cri-o://46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef" gracePeriod=25
Nov 03 00:45:05.228173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.228145    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:45:05.269640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.269610    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume\") pod \"18de8fc3-d694-4f27-935d-bef519be6005\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") "
Nov 03 00:45:05.269884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.269866    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume\") pod \"18de8fc3-d694-4f27-935d-bef519be6005\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") "
Nov 03 00:45:05.270264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.270248    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials\") pod \"18de8fc3-d694-4f27-935d-bef519be6005\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") "
Nov 03 00:45:05.270445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.270414    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp\") pod \"18de8fc3-d694-4f27-935d-bef519be6005\" (UID: \"18de8fc3-d694-4f27-935d-bef519be6005\") "
Nov 03 00:45:05.271282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.271253    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume" pod "18de8fc3-d694-4f27-935d-bef519be6005" (UID: "18de8fc3-d694-4f27-935d-bef519be6005"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:05.271423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.271398    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume" pod "18de8fc3-d694-4f27-935d-bef519be6005" (UID: "18de8fc3-d694-4f27-935d-bef519be6005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:05.274443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.273596    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials" pod "18de8fc3-d694-4f27-935d-bef519be6005" (UID: "18de8fc3-d694-4f27-935d-bef519be6005"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:05.274443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.273710    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp" pod "18de8fc3-d694-4f27-935d-bef519be6005" (UID: "18de8fc3-d694-4f27-935d-bef519be6005"). InnerVolumeSpecName "kube-api-access-69cmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:05.373169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.372603    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:05.373169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.372638    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cmp\" (UniqueName: \"kubernetes.io/projected/18de8fc3-d694-4f27-935d-bef519be6005-kube-api-access-69cmp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:05.373169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.372654    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:05.373169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.372670    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18de8fc3-d694-4f27-935d-bef519be6005-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:05.481598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.481559    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerStarted","Data":"dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f"}
Nov 03 00:45:05.483653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.483624    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerStarted","Data":"24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b"}
Nov 03 00:45:05.486504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.485925    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerStarted","Data":"4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7"}
Nov 03 00:45:05.488559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.488535    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerStarted","Data":"5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3"}
Nov 03 00:45:05.491288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.491206    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerStarted","Data":"7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee"}
Nov 03 00:45:05.493415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.493391    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerStarted","Data":"43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99"}
Nov 03 00:45:05.495415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.495391    4550 generic.go:356] "Generic (PLEG): container finished" podID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerID="c5d2000aecf5e84b78ff19fcfc781f739113f3686889702167e86c4283649b0b" exitCode=60
Nov 03 00:45:05.495533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.495457    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" event={"ID":"2cc1e19a-e169-4974-a452-3a9446f4b3ac","Type":"ContainerDied","Data":"c5d2000aecf5e84b78ff19fcfc781f739113f3686889702167e86c4283649b0b"}
Nov 03 00:45:05.495533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.495510    4550 scope.go:122] "RemoveContainer" containerID="9e272bac97a350133abe518ab05b22897d46d492cd4d1dc9d11ae6bceb2fd251"
Nov 03 00:45:05.495823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.495805    4550 scope.go:122] "RemoveContainer" containerID="c5d2000aecf5e84b78ff19fcfc781f739113f3686889702167e86c4283649b0b"
Nov 03 00:45:05.496116 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:05.496093    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz_odh-ai-gateway-infra(2cc1e19a-e169-4974-a452-3a9446f4b3ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac"
Nov 03 00:45:05.497791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.497769    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerStarted","Data":"1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5"}
Nov 03 00:45:05.499757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.499733    4550 generic.go:356] "Generic (PLEG): container finished" podID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerID="f060a4057d598658b00d9c4777f00a4dfde3020097e3bda546025c3b485ceb40" exitCode=60
Nov 03 00:45:05.499852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.499812    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerDied","Data":"f060a4057d598658b00d9c4777f00a4dfde3020097e3bda546025c3b485ceb40"}
Nov 03 00:45:05.500189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.500168    4550 scope.go:122] "RemoveContainer" containerID="f060a4057d598658b00d9c4777f00a4dfde3020097e3bda546025c3b485ceb40"
Nov 03 00:45:05.500514 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:05.500489    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w_odh-ai-gateway-infra(8d8abcb6-5936-438c-83a5-7050131bb1c5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5"
Nov 03 00:45:05.502124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.502104    4550 generic.go:356] "Generic (PLEG): container finished" podID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerID="9012ca3e4dfaa82901b1367f761232e96e79a9366bff27817fb7c4f2e251eac0" exitCode=60
Nov 03 00:45:05.502225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.502164    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerDied","Data":"9012ca3e4dfaa82901b1367f761232e96e79a9366bff27817fb7c4f2e251eac0"}
Nov 03 00:45:05.502642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.502621    4550 scope.go:122] "RemoveContainer" containerID="9012ca3e4dfaa82901b1367f761232e96e79a9366bff27817fb7c4f2e251eac0"
Nov 03 00:45:05.502913 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:05.502891    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx_odh-ai-gateway-infra(1bea83fd-c091-4b89-80e5-7728801df3f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1"
Nov 03 00:45:05.504349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.504326    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerStarted","Data":"a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83"}
Nov 03 00:45:05.506286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.506265    4550 generic.go:356] "Generic (PLEG): container finished" podID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerID="0d6af1eff00f3d6a290530ea48630496435b2e920b8b75edf48f87a47c9abeb7" exitCode=60
Nov 03 00:45:05.506451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.506334    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" event={"ID":"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db","Type":"ContainerDied","Data":"0d6af1eff00f3d6a290530ea48630496435b2e920b8b75edf48f87a47c9abeb7"}
Nov 03 00:45:05.506710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.506693    4550 scope.go:122] "RemoveContainer" containerID="0d6af1eff00f3d6a290530ea48630496435b2e920b8b75edf48f87a47c9abeb7"
Nov 03 00:45:05.506915 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:05.506893    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9_odh-ai-gateway-infra(e5c9eb1c-745a-403f-ae58-29e5b6c6d4db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db"
Nov 03 00:45:05.507725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.507707    4550 generic.go:356] "Generic (PLEG): container finished" podID="18de8fc3-d694-4f27-935d-bef519be6005" containerID="46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef" exitCode=0
Nov 03 00:45:05.507792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.507763    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7" event={"ID":"18de8fc3-d694-4f27-935d-bef519be6005","Type":"ContainerDied","Data":"46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef"}
Nov 03 00:45:05.507792 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.507788    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7" event={"ID":"18de8fc3-d694-4f27-935d-bef519be6005","Type":"ContainerDied","Data":"c4887d3e36dc3568ca72872a67e4b20e3cebede9936f6874e95df11f5383fa08"}
Nov 03 00:45:05.507856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.507836    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"
Nov 03 00:45:05.511659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.511638    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerStarted","Data":"d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec"}
Nov 03 00:45:05.511729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.511643    4550 scope.go:122] "RemoveContainer" containerID="13987fa7bd9986169038ad1f669215d3cb9f52dd43126d678662d8854266e4d8"
Nov 03 00:45:05.527402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.527052    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerStarted","Data":"d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20"}
Nov 03 00:45:05.529931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.529900    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerStarted","Data":"88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989"}
Nov 03 00:45:05.530372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.530198    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" podStartSLOduration=5.530180536 podStartE2EDuration="5.530180536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:05.522177319 +0000 UTC m=+2387.843714121" watchObservedRunningTime="2026-11-03 00:45:05.530180536 +0000 UTC m=+2387.851717336"
Nov 03 00:45:05.532113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.532088    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerStarted","Data":"84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b"}
Nov 03 00:45:05.538607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.538585    4550 scope.go:122] "RemoveContainer" containerID="f78c3246e4ad9935f203240e2d114ac699d1a7f228c21c27e6ebe84535e237a6"
Nov 03 00:45:05.552555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.552530    4550 scope.go:122] "RemoveContainer" containerID="0c5fc93363728e0f770805b2617ad73de578b132fa686e124cfb5481bf2c5048"
Nov 03 00:45:05.629889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.629863    4550 scope.go:122] "RemoveContainer" containerID="46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef"
Nov 03 00:45:05.639857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.639836    4550 scope.go:122] "RemoveContainer" containerID="46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef"
Nov 03 00:45:05.640151 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:05.640134    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef\": container with ID starting with 46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef not found: ID does not exist" containerID="46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef"
Nov 03 00:45:05.640204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.640163    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef"} err="failed to get container status \"46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef\": rpc error: code = NotFound desc = could not find container \"46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef\": container with ID starting with 46faf6dd969e9048dc6a44c74ca1dcb9ef01ec0ae465a42783327a12bd9273ef not found: ID does not exist"
Nov 03 00:45:05.931055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:05.931000    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" podStartSLOduration=5.930988216 podStartE2EDuration="5.930988216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:05.929885136 +0000 UTC m=+2388.251421934" watchObservedRunningTime="2026-11-03 00:45:05.930988216 +0000 UTC m=+2388.252525015"
Nov 03 00:45:06.014633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.014598    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"]
Nov 03 00:45:06.020952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.020918    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdbccw7"]
Nov 03 00:45:06.032702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.032677    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"]
Nov 03 00:45:06.036701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.036677    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2f4596-hg4g9"]
Nov 03 00:45:06.443933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.443899    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:06.548207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.548117    4550 generic.go:356] "Generic (PLEG): container finished" podID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerID="43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99" exitCode=60
Nov 03 00:45:06.548207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.548193    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerDied","Data":"43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99"}
Nov 03 00:45:06.548416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.548254    4550 scope.go:122] "RemoveContainer" containerID="54ebb7ded60cf7d76b09ccbeda6402d00a688f895337d83d62197aa8575738c6"
Nov 03 00:45:06.548648 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.548626    4550 scope.go:122] "RemoveContainer" containerID="43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99"
Nov 03 00:45:06.549033 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.549013    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk_odh-ai-gateway-infra(5201b966-0adb-46a4-bfbb-79edcd61db78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78"
Nov 03 00:45:06.552252 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.552229    4550 generic.go:356] "Generic (PLEG): container finished" podID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerID="1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5" exitCode=60
Nov 03 00:45:06.552371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.552282    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerDied","Data":"1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5"}
Nov 03 00:45:06.552677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.552660    4550 scope.go:122] "RemoveContainer" containerID="1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5"
Nov 03 00:45:06.553068 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.553041    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt_odh-ai-gateway-infra(e4ae6ba2-9455-4a12-a5ca-b5064d128f34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34"
Nov 03 00:45:06.560514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.560473    4550 generic.go:356] "Generic (PLEG): container finished" podID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerID="a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83" exitCode=60
Nov 03 00:45:06.560651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.560559    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerDied","Data":"a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83"}
Nov 03 00:45:06.560956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.560939    4550 scope.go:122] "RemoveContainer" containerID="a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83"
Nov 03 00:45:06.561336 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.561314    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp_odh-ai-gateway-infra(6437d6ba-9677-43f2-93f9-c9b1762799e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0"
Nov 03 00:45:06.564555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.564539    4550 scope.go:122] "RemoveContainer" containerID="8662e9619bdfbfd9f72270674e7e94b720a4b219afb61063f92ff8f96419dcd2"
Nov 03 00:45:06.566846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.566823    4550 generic.go:356] "Generic (PLEG): container finished" podID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerID="d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec" exitCode=60
Nov 03 00:45:06.566947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.566876    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerDied","Data":"d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec"}
Nov 03 00:45:06.567308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.567287    4550 scope.go:122] "RemoveContainer" containerID="d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec"
Nov 03 00:45:06.567779 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.567754    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q_odh-ai-gateway-infra(d91f4d59-9ff5-4054-8e0d-7701138e5f5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c"
Nov 03 00:45:06.568757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.568739    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerID="d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20" exitCode=60
Nov 03 00:45:06.568852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.568838    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerDied","Data":"d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20"}
Nov 03 00:45:06.569178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.569156    4550 scope.go:122] "RemoveContainer" containerID="d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20"
Nov 03 00:45:06.569471 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.569426    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg_odh-ai-gateway-infra(d9634e52-dcc3-4e14-b7ee-bfe7f207c870)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870"
Nov 03 00:45:06.571411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.571391    4550 generic.go:356] "Generic (PLEG): container finished" podID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerID="88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989" exitCode=60
Nov 03 00:45:06.571552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.571477    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerDied","Data":"88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989"}
Nov 03 00:45:06.571839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.571821    4550 scope.go:122] "RemoveContainer" containerID="88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989"
Nov 03 00:45:06.572105 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.572084    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk_odh-ai-gateway-infra(b758cbeb-2873-4a60-8781-ec74cd985a5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f"
Nov 03 00:45:06.574209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.574187    4550 generic.go:356] "Generic (PLEG): container finished" podID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerID="84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b" exitCode=60
Nov 03 00:45:06.574333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.574284    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerDied","Data":"84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b"}
Nov 03 00:45:06.574838 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.574669    4550 scope.go:122] "RemoveContainer" containerID="84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b"
Nov 03 00:45:06.574978 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.574956    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq_odh-ai-gateway-infra(a8014b63-70fd-4992-aa3d-7a6664cea4bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd"
Nov 03 00:45:06.576983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.576957    4550 generic.go:356] "Generic (PLEG): container finished" podID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerID="dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f" exitCode=60
Nov 03 00:45:06.577094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.577032    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerDied","Data":"dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f"}
Nov 03 00:45:06.577295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.577245    4550 scope.go:122] "RemoveContainer" containerID="dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f"
Nov 03 00:45:06.577585 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.577564    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn_odh-ai-gateway-infra(2044c984-2256-46c0-a939-bb5d39f2a9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0"
Nov 03 00:45:06.579672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.579652    4550 generic.go:356] "Generic (PLEG): container finished" podID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerID="24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b" exitCode=60
Nov 03 00:45:06.579786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.579727    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerDied","Data":"24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b"}
Nov 03 00:45:06.579952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.579930    4550 scope.go:122] "RemoveContainer" containerID="24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b"
Nov 03 00:45:06.580212 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.580193    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p_odh-ai-gateway-infra(ab7524e5-60fd-4d8a-8cac-75bd14006e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93"
Nov 03 00:45:06.581767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.581746    4550 generic.go:356] "Generic (PLEG): container finished" podID="2a596167-f57e-4c78-8d4f-d817b460d090" containerID="4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7" exitCode=60
Nov 03 00:45:06.581869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.581822    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerDied","Data":"4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7"}
Nov 03 00:45:06.582046 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.582034    4550 scope.go:122] "RemoveContainer" containerID="4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7"
Nov 03 00:45:06.582311 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.582241    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq_odh-ai-gateway-infra(2a596167-f57e-4c78-8d4f-d817b460d090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" podUID="2a596167-f57e-4c78-8d4f-d817b460d090"
Nov 03 00:45:06.582414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.582394    4550 scope.go:122] "RemoveContainer" containerID="7deeb323b8bc08bed811c40925e3fad0c34be868463bd1dc503c3f8269cd3028"
Nov 03 00:45:06.585054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.585033    4550 generic.go:356] "Generic (PLEG): container finished" podID="56666705-dfb5-4876-b27b-22ee10e32c11" containerID="5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3" exitCode=60
Nov 03 00:45:06.585163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.585114    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerDied","Data":"5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3"}
Nov 03 00:45:06.585507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.585489    4550 scope.go:122] "RemoveContainer" containerID="5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3"
Nov 03 00:45:06.585831 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.585808    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8_odh-ai-gateway-infra(56666705-dfb5-4876-b27b-22ee10e32c11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" podUID="56666705-dfb5-4876-b27b-22ee10e32c11"
Nov 03 00:45:06.587710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.587691    4550 generic.go:356] "Generic (PLEG): container finished" podID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerID="7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee" exitCode=60
Nov 03 00:45:06.587796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.587766    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerDied","Data":"7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee"}
Nov 03 00:45:06.587990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.587975    4550 scope.go:122] "RemoveContainer" containerID="7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee"
Nov 03 00:45:06.588274 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:06.588239    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj_odh-ai-gateway-infra(7d9a0a98-a505-4165-be89-7b45f49226e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8"
Nov 03 00:45:06.599975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.599953    4550 scope.go:122] "RemoveContainer" containerID="fe7cfd7b6acee1160aa02cfa93259ab4536236ea4fda03805598df07fa0ee6d7"
Nov 03 00:45:06.614114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.614066    4550 scope.go:122] "RemoveContainer" containerID="e4139d152a5ca5c3705ba3d8266073a038fdff39dc580efc36eb8a624238d8c7"
Nov 03 00:45:06.630182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.630159    4550 scope.go:122] "RemoveContainer" containerID="5609ac200709eb42bb0d803f920658981b0ee61c3186b28e964730b55f611612"
Nov 03 00:45:06.644713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.644688    4550 scope.go:122] "RemoveContainer" containerID="f7154f0c55b1c417b93d053a6927c9bcacc53b5cf777f59006d53223f7a12080"
Nov 03 00:45:06.659347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.659323    4550 scope.go:122] "RemoveContainer" containerID="35a98163d3843b12f0b80886e06d9d4f026c8889d717ba856767d375ef74d021"
Nov 03 00:45:06.675517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.675497    4550 scope.go:122] "RemoveContainer" containerID="ced70645927abfa36bd5d39f6c09655bf6c950413bc317ce12eddd2055df4c10"
Nov 03 00:45:06.690099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.690076    4550 scope.go:122] "RemoveContainer" containerID="962810a61c4c350acb50bc36097a8627439f9c2f4ab3bb07df8b10365adcd825"
Nov 03 00:45:06.706655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.706637    4550 scope.go:122] "RemoveContainer" containerID="4f728c72168d2931935f2d972f9a20d04cd6a9d3b75d39c178c14048edc9ab92"
Nov 03 00:45:06.721168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:06.721146    4550 scope.go:122] "RemoveContainer" containerID="47d30963176408d7ebed5ea64de121a26a1a1f4c8695b9fb7744e09996d4263c"
Nov 03 00:45:07.422367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.422328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:07.520970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.520920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:07.597609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.597570    4550 scope.go:122] "RemoveContainer" containerID="1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5"
Nov 03 00:45:07.598027 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.597999    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt_odh-ai-gateway-infra(e4ae6ba2-9455-4a12-a5ca-b5064d128f34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34"
Nov 03 00:45:07.599416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.599391    4550 scope.go:122] "RemoveContainer" containerID="a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83"
Nov 03 00:45:07.599802 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.599780    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp_odh-ai-gateway-infra(6437d6ba-9677-43f2-93f9-c9b1762799e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0"
Nov 03 00:45:07.601015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.600994    4550 scope.go:122] "RemoveContainer" containerID="d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec"
Nov 03 00:45:07.601339 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.601319    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q_odh-ai-gateway-infra(d91f4d59-9ff5-4054-8e0d-7701138e5f5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c"
Nov 03 00:45:07.602586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.602565    4550 scope.go:122] "RemoveContainer" containerID="d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20"
Nov 03 00:45:07.602861 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.602825    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg_odh-ai-gateway-infra(d9634e52-dcc3-4e14-b7ee-bfe7f207c870)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870"
Nov 03 00:45:07.604295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.604278    4550 scope.go:122] "RemoveContainer" containerID="88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989"
Nov 03 00:45:07.604546 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.604527    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk_odh-ai-gateway-infra(b758cbeb-2873-4a60-8781-ec74cd985a5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f"
Nov 03 00:45:07.606118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.606098    4550 scope.go:122] "RemoveContainer" containerID="84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b"
Nov 03 00:45:07.606402 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.606380    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq_odh-ai-gateway-infra(a8014b63-70fd-4992-aa3d-7a6664cea4bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd"
Nov 03 00:45:07.609490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.607635    4550 scope.go:122] "RemoveContainer" containerID="dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f"
Nov 03 00:45:07.609490 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.607894    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn_odh-ai-gateway-infra(2044c984-2256-46c0-a939-bb5d39f2a9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0"
Nov 03 00:45:07.613676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.613654    4550 scope.go:122] "RemoveContainer" containerID="24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b"
Nov 03 00:45:07.613965 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.613937    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p_odh-ai-gateway-infra(ab7524e5-60fd-4d8a-8cac-75bd14006e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93"
Nov 03 00:45:07.615326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.615307    4550 scope.go:122] "RemoveContainer" containerID="4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7"
Nov 03 00:45:07.615614 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.615582    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq_odh-ai-gateway-infra(2a596167-f57e-4c78-8d4f-d817b460d090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" podUID="2a596167-f57e-4c78-8d4f-d817b460d090"
Nov 03 00:45:07.616970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.616949    4550 scope.go:122] "RemoveContainer" containerID="5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3"
Nov 03 00:45:07.617212 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.617174    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8_odh-ai-gateway-infra(56666705-dfb5-4876-b27b-22ee10e32c11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" podUID="56666705-dfb5-4876-b27b-22ee10e32c11"
Nov 03 00:45:07.618529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.618511    4550 scope.go:122] "RemoveContainer" containerID="7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee"
Nov 03 00:45:07.618741 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.618722    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj_odh-ai-gateway-infra(7d9a0a98-a505-4165-be89-7b45f49226e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8"
Nov 03 00:45:07.619995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.619980    4550 scope.go:122] "RemoveContainer" containerID="43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99"
Nov 03 00:45:07.620156 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:07.620142    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk_odh-ai-gateway-infra(5201b966-0adb-46a4-bfbb-79edcd61db78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78"
Nov 03 00:45:07.812989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.812913    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18de8fc3-d694-4f27-935d-bef519be6005" path="/var/lib/kubelet/pods/18de8fc3-d694-4f27-935d-bef519be6005/volumes"
Nov 03 00:45:07.813421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:07.813407    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5a33ee-80c4-4654-b704-a154afb81ad3" path="/var/lib/kubelet/pods/6f5a33ee-80c4-4654-b704-a154afb81ad3/volumes"
Nov 03 00:45:08.232593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:08.232562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:09.822978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:09.822933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:09.927644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:09.927595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.087237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:11.087195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.187584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:11.187551    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.505502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:11.505468    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.641487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:11.641450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.742674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:11.742641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:11.898152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:11.898070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:12.012246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:12.012213    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:12.112072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:12.112039    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:12.531883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:12.531484    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:12.629532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:12.629491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:13.507872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:13.507838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:13.791047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:13.790942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:15.336057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:15.336017    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:16.034007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.033969    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"]
Nov 03 00:45:16.046249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.046217    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82b8179f-27nwp"]
Nov 03 00:45:16.802194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.802156    4550 scope.go:122] "RemoveContainer" containerID="88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989"
Nov 03 00:45:16.802923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.802258    4550 scope.go:122] "RemoveContainer" containerID="0d6af1eff00f3d6a290530ea48630496435b2e920b8b75edf48f87a47c9abeb7"
Nov 03 00:45:16.803018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.802979    4550 scope.go:122] "RemoveContainer" containerID="5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3"
Nov 03 00:45:16.803277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.803258    4550 scope.go:122] "RemoveContainer" containerID="c5d2000aecf5e84b78ff19fcfc781f739113f3686889702167e86c4283649b0b"
Nov 03 00:45:16.804411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.803884    4550 scope.go:122] "RemoveContainer" containerID="43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99"
Nov 03 00:45:16.804502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.804468    4550 scope.go:122] "RemoveContainer" containerID="7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee"
Nov 03 00:45:16.981821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:16.981793    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:17.699350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.699317    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerStarted","Data":"0b2b8be20b9f69bbacb7a43a6b50c39b58b8cddda192039789f8f48c7b8a21a7"}
Nov 03 00:45:17.701564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.701532    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" event={"ID":"2cc1e19a-e169-4974-a452-3a9446f4b3ac","Type":"ContainerStarted","Data":"e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059"}
Nov 03 00:45:17.703783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.703757    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" event={"ID":"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db","Type":"ContainerStarted","Data":"1b684983f0538770299a69efe2a69b426cd1ca7d137cb63d3a17fe1aee4195d9"}
Nov 03 00:45:17.705988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.705955    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerStarted","Data":"c5a0935f1889a3c305d316c70465a871b0421eb68bb56611e4ae23c0400fe193"}
Nov 03 00:45:17.707758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.707736    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerStarted","Data":"a0ad7974f322d4121777b31e754097ed8d2b8f883e85815fdac7c13a6208630f"}
Nov 03 00:45:17.709562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.709541    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerStarted","Data":"4ca012396f3726407b5a873e54bb9cfb6d15e63fa02b247d45f476abed4a1b68"}
Nov 03 00:45:17.801274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.801182    4550 scope.go:122] "RemoveContainer" containerID="a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83"
Nov 03 00:45:17.802328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.801492    4550 scope.go:122] "RemoveContainer" containerID="1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5"
Nov 03 00:45:17.813698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.813668    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a710bd-7cfd-4afa-a338-641fb96ca261" path="/var/lib/kubelet/pods/20a710bd-7cfd-4afa-a338-641fb96ca261/volumes"
Nov 03 00:45:17.853737 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.853705    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"]
Nov 03 00:45:17.895228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:17.895199    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"]
Nov 03 00:45:18.265274 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:18.265235    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c9eb1c_745a_403f_ae58_29e5b6c6d4db.slice/crio-conmon-1b684983f0538770299a69efe2a69b426cd1ca7d137cb63d3a17fe1aee4195d9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:18.719735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.719702    4550 generic.go:356] "Generic (PLEG): container finished" podID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerID="c5a0935f1889a3c305d316c70465a871b0421eb68bb56611e4ae23c0400fe193" exitCode=60
Nov 03 00:45:18.719977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.719788    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerDied","Data":"c5a0935f1889a3c305d316c70465a871b0421eb68bb56611e4ae23c0400fe193"}
Nov 03 00:45:18.719977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.719885    4550 scope.go:122] "RemoveContainer" containerID="88844494d1b862e67f9a47232a04ad010fc7150154ee9df3c8a0bf5b77f11989"
Nov 03 00:45:18.720356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.720335    4550 scope.go:122] "RemoveContainer" containerID="c5a0935f1889a3c305d316c70465a871b0421eb68bb56611e4ae23c0400fe193"
Nov 03 00:45:18.720742 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:18.720717    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk_odh-ai-gateway-infra(b758cbeb-2873-4a60-8781-ec74cd985a5f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f"
Nov 03 00:45:18.721897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.721877    4550 generic.go:356] "Generic (PLEG): container finished" podID="56666705-dfb5-4876-b27b-22ee10e32c11" containerID="a0ad7974f322d4121777b31e754097ed8d2b8f883e85815fdac7c13a6208630f" exitCode=60
Nov 03 00:45:18.721987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.721942    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerDied","Data":"a0ad7974f322d4121777b31e754097ed8d2b8f883e85815fdac7c13a6208630f"}
Nov 03 00:45:18.722338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.722321    4550 scope.go:122] "RemoveContainer" containerID="a0ad7974f322d4121777b31e754097ed8d2b8f883e85815fdac7c13a6208630f"
Nov 03 00:45:18.722670 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:18.722647    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8_odh-ai-gateway-infra(56666705-dfb5-4876-b27b-22ee10e32c11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" podUID="56666705-dfb5-4876-b27b-22ee10e32c11"
Nov 03 00:45:18.723899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.723881    4550 generic.go:356] "Generic (PLEG): container finished" podID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerID="4ca012396f3726407b5a873e54bb9cfb6d15e63fa02b247d45f476abed4a1b68" exitCode=60
Nov 03 00:45:18.723982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.723949    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerDied","Data":"4ca012396f3726407b5a873e54bb9cfb6d15e63fa02b247d45f476abed4a1b68"}
Nov 03 00:45:18.724346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.724322    4550 scope.go:122] "RemoveContainer" containerID="4ca012396f3726407b5a873e54bb9cfb6d15e63fa02b247d45f476abed4a1b68"
Nov 03 00:45:18.724660 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:18.724638    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj_odh-ai-gateway-infra(7d9a0a98-a505-4165-be89-7b45f49226e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8"
Nov 03 00:45:18.726055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.726038    4550 generic.go:356] "Generic (PLEG): container finished" podID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerID="0b2b8be20b9f69bbacb7a43a6b50c39b58b8cddda192039789f8f48c7b8a21a7" exitCode=60
Nov 03 00:45:18.726138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.726111    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerDied","Data":"0b2b8be20b9f69bbacb7a43a6b50c39b58b8cddda192039789f8f48c7b8a21a7"}
Nov 03 00:45:18.726531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.726509    4550 scope.go:122] "RemoveContainer" containerID="0b2b8be20b9f69bbacb7a43a6b50c39b58b8cddda192039789f8f48c7b8a21a7"
Nov 03 00:45:18.726822 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:18.726799    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk_odh-ai-gateway-infra(5201b966-0adb-46a4-bfbb-79edcd61db78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78"
Nov 03 00:45:18.727899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.727880    4550 generic.go:356] "Generic (PLEG): container finished" podID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerID="e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059" exitCode=60
Nov 03 00:45:18.727980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.727960    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" event={"ID":"2cc1e19a-e169-4974-a452-3a9446f4b3ac","Type":"ContainerDied","Data":"e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059"}
Nov 03 00:45:18.730133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.730111    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerStarted","Data":"db0f6dc07f982a4851609eac020caf12a1077d56eac53efd743adfa5db0e818f"}
Nov 03 00:45:18.732221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.732201    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerStarted","Data":"1a9eb0a0cf485cef36c44bf54baef23c95d9b18fccd9798657dc22bdb04e5438"}
Nov 03 00:45:18.734223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.734204    4550 generic.go:356] "Generic (PLEG): container finished" podID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerID="1b684983f0538770299a69efe2a69b426cd1ca7d137cb63d3a17fe1aee4195d9" exitCode=60
Nov 03 00:45:18.734311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.734282    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" event={"ID":"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db","Type":"ContainerDied","Data":"1b684983f0538770299a69efe2a69b426cd1ca7d137cb63d3a17fe1aee4195d9"}
Nov 03 00:45:18.735121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.735098    4550 scope.go:122] "RemoveContainer" containerID="5932509c6b109ea720771694ba1835a166088af41fb7f63f97965315eecbd1b3"
Nov 03 00:45:18.754337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.754310    4550 scope.go:122] "RemoveContainer" containerID="7969dfe840b339a92cbe44df55cde13af265f2adfdb20223d39532b080341eee"
Nov 03 00:45:18.758421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.757998    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"]
Nov 03 00:45:18.820116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.819702    4550 scope.go:122] "RemoveContainer" containerID="43abc5576c7c9f8161b2fd7f74edb49035d696c0da274c654db768c6d5b5cb99"
Nov 03 00:45:18.834292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.823967    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:18.834292 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.831557    4550 scope.go:122] "RemoveContainer" containerID="d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec"
Nov 03 00:45:18.845588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.845558    4550 scope.go:122] "RemoveContainer" containerID="d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20"
Nov 03 00:45:18.872535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.866061    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"]
Nov 03 00:45:18.892751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.892724    4550 scope.go:122] "RemoveContainer" containerID="c5d2000aecf5e84b78ff19fcfc781f739113f3686889702167e86c4283649b0b"
Nov 03 00:45:18.921504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.920899    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"]
Nov 03 00:45:18.971303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:18.971218    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"]
Nov 03 00:45:19.003800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.003777    4550 scope.go:122] "RemoveContainer" containerID="0d6af1eff00f3d6a290530ea48630496435b2e920b8b75edf48f87a47c9abeb7"
Nov 03 00:45:19.065980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.065955    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:19.068209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.068187    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:19.128820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.128785    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv\" (UniqueName: \"kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv\") pod \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") "
Nov 03 00:45:19.128985 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.128870    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca\") pod \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") "
Nov 03 00:45:19.129052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.129021    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb\" (UniqueName: \"kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb\") pod \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\" (UID: \"2cc1e19a-e169-4974-a452-3a9446f4b3ac\") "
Nov 03 00:45:19.129121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.129102    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca\") pod \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\" (UID: \"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db\") "
Nov 03 00:45:19.129254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.129225    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca" pod "2cc1e19a-e169-4974-a452-3a9446f4b3ac" (UID: "2cc1e19a-e169-4974-a452-3a9446f4b3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:19.129515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.129491    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca" pod "e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" (UID: "e5c9eb1c-745a-403f-ae58-29e5b6c6d4db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:19.129733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.129717    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:19.129787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.129735    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc1e19a-e169-4974-a452-3a9446f4b3ac-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:19.131611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.131584    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb" pod "2cc1e19a-e169-4974-a452-3a9446f4b3ac" (UID: "2cc1e19a-e169-4974-a452-3a9446f4b3ac"). InnerVolumeSpecName "kube-api-access-jz4fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:19.131728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.131637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv" pod "e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" (UID: "e5c9eb1c-745a-403f-ae58-29e5b6c6d4db"). InnerVolumeSpecName "kube-api-access-gvhsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:19.231631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.231547    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhsv\" (UniqueName: \"kubernetes.io/projected/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db-kube-api-access-gvhsv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:19.231631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.231579    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4fb\" (UniqueName: \"kubernetes.io/projected/2cc1e19a-e169-4974-a452-3a9446f4b3ac-kube-api-access-jz4fb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:19.742894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.742855    4550 generic.go:356] "Generic (PLEG): container finished" podID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerID="db0f6dc07f982a4851609eac020caf12a1077d56eac53efd743adfa5db0e818f" exitCode=60
Nov 03 00:45:19.743107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.742926    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerDied","Data":"db0f6dc07f982a4851609eac020caf12a1077d56eac53efd743adfa5db0e818f"}
Nov 03 00:45:19.743107 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.742983    4550 scope.go:122] "RemoveContainer" containerID="1936bfbeaf67470dbae46536f776d742a1df43e58952e461cc5bd4594e4ee9d5"
Nov 03 00:45:19.743592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.743405    4550 scope.go:122] "RemoveContainer" containerID="db0f6dc07f982a4851609eac020caf12a1077d56eac53efd743adfa5db0e818f"
Nov 03 00:45:19.743785 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:19.743761    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt_odh-ai-gateway-infra(e4ae6ba2-9455-4a12-a5ca-b5064d128f34)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34"
Nov 03 00:45:19.745035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.745013    4550 generic.go:356] "Generic (PLEG): container finished" podID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerID="1a9eb0a0cf485cef36c44bf54baef23c95d9b18fccd9798657dc22bdb04e5438" exitCode=60
Nov 03 00:45:19.745116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.745083    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerDied","Data":"1a9eb0a0cf485cef36c44bf54baef23c95d9b18fccd9798657dc22bdb04e5438"}
Nov 03 00:45:19.745445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.745416    4550 scope.go:122] "RemoveContainer" containerID="1a9eb0a0cf485cef36c44bf54baef23c95d9b18fccd9798657dc22bdb04e5438"
Nov 03 00:45:19.745688 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:19.745668    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp_odh-ai-gateway-infra(6437d6ba-9677-43f2-93f9-c9b1762799e0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0"
Nov 03 00:45:19.746378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.746360    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"
Nov 03 00:45:19.746378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.746364    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9" event={"ID":"e5c9eb1c-745a-403f-ae58-29e5b6c6d4db","Type":"ContainerDied","Data":"ac5216cf3537b9088c3ad7a80bb4213a28130bb626109df84a50d38ffe50d6b7"}
Nov 03 00:45:19.748487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.748466    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerStarted","Data":"cbd777d33ead7efca6dd5a20cad28f90c4d7597bfb2f29f441839e07e7390598"}
Nov 03 00:45:19.750615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.750596    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerStarted","Data":"08d603ddc278c8173730963f9f6b49c75cff4b34feb9e4453e6119115252c4de"}
Nov 03 00:45:19.759965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.759941    4550 scope.go:122] "RemoveContainer" containerID="a5ff226f2ace6a68431b5c1fffdc9e3ca8a0de3e12853e5ac00e7ab739eeeb83"
Nov 03 00:45:19.761940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.761915    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz" event={"ID":"2cc1e19a-e169-4974-a452-3a9446f4b3ac","Type":"ContainerDied","Data":"327902daf65befedbaa8aa45db71095da127bf22713e271f8bb033d4d903b507"}
Nov 03 00:45:19.762047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.762003    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"
Nov 03 00:45:19.787101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.787075    4550 scope.go:122] "RemoveContainer" containerID="1b684983f0538770299a69efe2a69b426cd1ca7d137cb63d3a17fe1aee4195d9"
Nov 03 00:45:19.804312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.804277    4550 scope.go:122] "RemoveContainer" containerID="f060a4057d598658b00d9c4777f00a4dfde3020097e3bda546025c3b485ceb40"
Nov 03 00:45:19.805242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.804583    4550 scope.go:122] "RemoveContainer" containerID="24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b"
Nov 03 00:45:19.805242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.804933    4550 scope.go:122] "RemoveContainer" containerID="dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f"
Nov 03 00:45:19.805628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.805547    4550 scope.go:122] "RemoveContainer" containerID="9012ca3e4dfaa82901b1367f761232e96e79a9366bff27817fb7c4f2e251eac0"
Nov 03 00:45:19.806594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.805897    4550 scope.go:122] "RemoveContainer" containerID="4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7"
Nov 03 00:45:19.813798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:19.812259    4550 scope.go:122] "RemoveContainer" containerID="e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059"
Nov 03 00:45:20.062138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.062109    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"]
Nov 03 00:45:20.070870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.070831    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894445-xjlp9"]
Nov 03 00:45:20.116074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.115494    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"]
Nov 03 00:45:20.131807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.131725    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"]
Nov 03 00:45:20.135831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.135807    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894445-bnpnz"]
Nov 03 00:45:20.141235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.141212    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"]
Nov 03 00:45:20.146145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.146058    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:20.258225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.258197    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r\" (UniqueName: \"kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r\") pod \"b758cbeb-2873-4a60-8781-ec74cd985a5f\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") "
Nov 03 00:45:20.258512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.258490    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca\") pod \"b758cbeb-2873-4a60-8781-ec74cd985a5f\" (UID: \"b758cbeb-2873-4a60-8781-ec74cd985a5f\") "
Nov 03 00:45:20.259069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.258893    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca" pod "b758cbeb-2873-4a60-8781-ec74cd985a5f" (UID: "b758cbeb-2873-4a60-8781-ec74cd985a5f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:20.259307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.259285    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b758cbeb-2873-4a60-8781-ec74cd985a5f-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.261655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.261631    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r" pod "b758cbeb-2873-4a60-8781-ec74cd985a5f" (UID: "b758cbeb-2873-4a60-8781-ec74cd985a5f"). InnerVolumeSpecName "kube-api-access-mlx4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:20.360417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.360344    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlx4r\" (UniqueName: \"kubernetes.io/projected/b758cbeb-2873-4a60-8781-ec74cd985a5f-kube-api-access-mlx4r\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.386291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.386259    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:20.392581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.392556    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:20.394622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.394602    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:20.461614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.461573    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca\") pod \"56666705-dfb5-4876-b27b-22ee10e32c11\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") "
Nov 03 00:45:20.461827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.461623    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn\" (UniqueName: \"kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn\") pod \"7d9a0a98-a505-4165-be89-7b45f49226e8\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") "
Nov 03 00:45:20.461827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.461666    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp\" (UniqueName: \"kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp\") pod \"56666705-dfb5-4876-b27b-22ee10e32c11\" (UID: \"56666705-dfb5-4876-b27b-22ee10e32c11\") "
Nov 03 00:45:20.461827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.461707    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca\") pod \"5201b966-0adb-46a4-bfbb-79edcd61db78\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") "
Nov 03 00:45:20.461827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.461776    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca\") pod \"7d9a0a98-a505-4165-be89-7b45f49226e8\" (UID: \"7d9a0a98-a505-4165-be89-7b45f49226e8\") "
Nov 03 00:45:20.462186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.461903    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47\" (UniqueName: \"kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47\") pod \"5201b966-0adb-46a4-bfbb-79edcd61db78\" (UID: \"5201b966-0adb-46a4-bfbb-79edcd61db78\") "
Nov 03 00:45:20.462744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.462698    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca" pod "7d9a0a98-a505-4165-be89-7b45f49226e8" (UID: "7d9a0a98-a505-4165-be89-7b45f49226e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:20.464009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.463982    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca" pod "56666705-dfb5-4876-b27b-22ee10e32c11" (UID: "56666705-dfb5-4876-b27b-22ee10e32c11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:20.464560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.464509    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca" pod "5201b966-0adb-46a4-bfbb-79edcd61db78" (UID: "5201b966-0adb-46a4-bfbb-79edcd61db78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:20.532072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.532026    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:20.557525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.556603    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp" pod "56666705-dfb5-4876-b27b-22ee10e32c11" (UID: "56666705-dfb5-4876-b27b-22ee10e32c11"). InnerVolumeSpecName "kube-api-access-46kbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:20.557525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.556807    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn" pod "7d9a0a98-a505-4165-be89-7b45f49226e8" (UID: "7d9a0a98-a505-4165-be89-7b45f49226e8"). InnerVolumeSpecName "kube-api-access-t2szn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:20.559682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.557937    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47" pod "5201b966-0adb-46a4-bfbb-79edcd61db78" (UID: "5201b966-0adb-46a4-bfbb-79edcd61db78"). InnerVolumeSpecName "kube-api-access-frm47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:20.563757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.563730    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56666705-dfb5-4876-b27b-22ee10e32c11-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.563757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.563759    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2szn\" (UniqueName: \"kubernetes.io/projected/7d9a0a98-a505-4165-be89-7b45f49226e8-kube-api-access-t2szn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.563918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.563773    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46kbp\" (UniqueName: \"kubernetes.io/projected/56666705-dfb5-4876-b27b-22ee10e32c11-kube-api-access-46kbp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.563918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.563785    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5201b966-0adb-46a4-bfbb-79edcd61db78-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.563918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.563796    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d9a0a98-a505-4165-be89-7b45f49226e8-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.563918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.563809    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frm47\" (UniqueName: \"kubernetes.io/projected/5201b966-0adb-46a4-bfbb-79edcd61db78-kube-api-access-frm47\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:20.773944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.773909    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"
Nov 03 00:45:20.773944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.773915    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk" event={"ID":"5201b966-0adb-46a4-bfbb-79edcd61db78","Type":"ContainerDied","Data":"0ebcd0d7c6f1ee8d5051421c8cb4bb4aadd11ec206b862fbbb296eeb9048c28a"}
Nov 03 00:45:20.774176 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.773981    4550 scope.go:122] "RemoveContainer" containerID="0b2b8be20b9f69bbacb7a43a6b50c39b58b8cddda192039789f8f48c7b8a21a7"
Nov 03 00:45:20.779785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.779753    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerStarted","Data":"528af1f5f213543894531ddb410d92fd37a990a546715e2e777d26e52009c5b8"}
Nov 03 00:45:20.782084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.782052    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerStarted","Data":"f393bbf964cbc1313e99540b7e06906c72254bbd02800f007e1123f73225bf1e"}
Nov 03 00:45:20.788376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.788356    4550 generic.go:356] "Generic (PLEG): container finished" podID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerID="cbd777d33ead7efca6dd5a20cad28f90c4d7597bfb2f29f441839e07e7390598" exitCode=60
Nov 03 00:45:20.788492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.788476    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerDied","Data":"cbd777d33ead7efca6dd5a20cad28f90c4d7597bfb2f29f441839e07e7390598"}
Nov 03 00:45:20.788561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.788526    4550 scope.go:122] "RemoveContainer" containerID="d9802e3499a82b33fc4cace2ed2c9376d29e86ac5961dfa741dcf8275cdc6dec"
Nov 03 00:45:20.788959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.788943    4550 scope.go:122] "RemoveContainer" containerID="cbd777d33ead7efca6dd5a20cad28f90c4d7597bfb2f29f441839e07e7390598"
Nov 03 00:45:20.789298 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:20.789277    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q_odh-ai-gateway-infra(d91f4d59-9ff5-4054-8e0d-7701138e5f5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c"
Nov 03 00:45:20.792974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.792953    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerID="08d603ddc278c8173730963f9f6b49c75cff4b34feb9e4453e6119115252c4de" exitCode=60
Nov 03 00:45:20.793081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.793056    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerDied","Data":"08d603ddc278c8173730963f9f6b49c75cff4b34feb9e4453e6119115252c4de"}
Nov 03 00:45:20.793799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.793446    4550 scope.go:122] "RemoveContainer" containerID="08d603ddc278c8173730963f9f6b49c75cff4b34feb9e4453e6119115252c4de"
Nov 03 00:45:20.793799 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:20.793743    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg_odh-ai-gateway-infra(d9634e52-dcc3-4e14-b7ee-bfe7f207c870)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870"
Nov 03 00:45:20.795979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.795722    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"
Nov 03 00:45:20.795979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.795724    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk" event={"ID":"b758cbeb-2873-4a60-8781-ec74cd985a5f","Type":"ContainerDied","Data":"c2065faf24e7df700e17e5b2ac15584187d0c4a15a10e624defc59d4e9f8af66"}
Nov 03 00:45:20.797790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.797766    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerStarted","Data":"5179a168f8173d31fdb5948fcdd32b8b8e5dcf25d81db1d260a5a5d542e424ce"}
Nov 03 00:45:20.800810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.800672    4550 scope.go:122] "RemoveContainer" containerID="84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b"
Nov 03 00:45:20.801529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.801353    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerStarted","Data":"798391ae0513ec9b491d0766edfacc96264d80f4b5fee4d26fea5868a6c2ca27"}
Nov 03 00:45:20.803962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.803938    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerStarted","Data":"ea58193d2ad6a17e15b3c68f5aa4d82c0dea4e14d68ba9e9e1b977121b033e96"}
Nov 03 00:45:20.806618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.806597    4550 scope.go:122] "RemoveContainer" containerID="d2c6f672b9ecb50efb386d39869155a1c1de2ff8678bd4496270c8a84788bd20"
Nov 03 00:45:20.808100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.808080    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"
Nov 03 00:45:20.808799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.808769    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8" event={"ID":"56666705-dfb5-4876-b27b-22ee10e32c11","Type":"ContainerDied","Data":"5bcfd8478b41dfefbac6c92bab286348a559a21d478f254573fd8036e3b99ec7"}
Nov 03 00:45:20.811826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.811801    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj" event={"ID":"7d9a0a98-a505-4165-be89-7b45f49226e8","Type":"ContainerDied","Data":"84a60ada027f66af4f0f15c1274a676aa05dca293ae59496beda7b1e47ced2bc"}
Nov 03 00:45:20.811922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.811899    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"
Nov 03 00:45:20.831262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.831237    4550 scope.go:122] "RemoveContainer" containerID="c5a0935f1889a3c305d316c70465a871b0421eb68bb56611e4ae23c0400fe193"
Nov 03 00:45:20.837341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.837250    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"]
Nov 03 00:45:20.879351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.878629    4550 scope.go:122] "RemoveContainer" containerID="a0ad7974f322d4121777b31e754097ed8d2b8f883e85815fdac7c13a6208630f"
Nov 03 00:45:20.885660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.885612    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"]
Nov 03 00:45:20.893264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.893206    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894445-txpmk"]
Nov 03 00:45:20.899823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.899789    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"]
Nov 03 00:45:20.907686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:20.907659    4550 scope.go:122] "RemoveContainer" containerID="4ca012396f3726407b5a873e54bb9cfb6d15e63fa02b247d45f476abed4a1b68"
Nov 03 00:45:21.026479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.026445    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:21.077716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.077687    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:21.082091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.082060    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr\" (UniqueName: \"kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr\") pod \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") "
Nov 03 00:45:21.082256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.082111    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca\") pod \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\" (UID: \"e4ae6ba2-9455-4a12-a5ca-b5064d128f34\") "
Nov 03 00:45:21.082482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.082460    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca" pod "e4ae6ba2-9455-4a12-a5ca-b5064d128f34" (UID: "e4ae6ba2-9455-4a12-a5ca-b5064d128f34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:21.082889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.082869    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:21.084134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.084113    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr" pod "e4ae6ba2-9455-4a12-a5ca-b5064d128f34" (UID: "e4ae6ba2-9455-4a12-a5ca-b5064d128f34"). InnerVolumeSpecName "kube-api-access-8bkzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:21.183810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.183727    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca\") pod \"6437d6ba-9677-43f2-93f9-c9b1762799e0\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") "
Nov 03 00:45:21.183810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.183803    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd\") pod \"6437d6ba-9677-43f2-93f9-c9b1762799e0\" (UID: \"6437d6ba-9677-43f2-93f9-c9b1762799e0\") "
Nov 03 00:45:21.184150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.184126    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca" pod "6437d6ba-9677-43f2-93f9-c9b1762799e0" (UID: "6437d6ba-9677-43f2-93f9-c9b1762799e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:21.184359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.184338    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bkzr\" (UniqueName: \"kubernetes.io/projected/e4ae6ba2-9455-4a12-a5ca-b5064d128f34-kube-api-access-8bkzr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:21.184359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.184359    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6437d6ba-9677-43f2-93f9-c9b1762799e0-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:21.208091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.208059    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"]
Nov 03 00:45:21.211885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.211857    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894445-9wzxj"]
Nov 03 00:45:21.245966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.245931    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"]
Nov 03 00:45:21.250601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.250567    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894445-w5nbk"]
Nov 03 00:45:21.258479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.257973    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd" pod "6437d6ba-9677-43f2-93f9-c9b1762799e0" (UID: "6437d6ba-9677-43f2-93f9-c9b1762799e0"). InnerVolumeSpecName "kube-api-access-5tlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:21.261905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.261142    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"]
Nov 03 00:45:21.284773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.284743    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"]
Nov 03 00:45:21.285330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.285310    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlpd\" (UniqueName: \"kubernetes.io/projected/6437d6ba-9677-43f2-93f9-c9b1762799e0-kube-api-access-5tlpd\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:21.294529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.294499    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894445-dzxx8"]
Nov 03 00:45:21.298259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.298232    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"]
Nov 03 00:45:21.813734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.813697    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" path="/var/lib/kubelet/pods/2cc1e19a-e169-4974-a452-3a9446f4b3ac/volumes"
Nov 03 00:45:21.814333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.814316    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" path="/var/lib/kubelet/pods/5201b966-0adb-46a4-bfbb-79edcd61db78/volumes"
Nov 03 00:45:21.814769 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.814754    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" path="/var/lib/kubelet/pods/56666705-dfb5-4876-b27b-22ee10e32c11/volumes"
Nov 03 00:45:21.815459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.815440    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" path="/var/lib/kubelet/pods/7d9a0a98-a505-4165-be89-7b45f49226e8/volumes"
Nov 03 00:45:21.815964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.815946    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" path="/var/lib/kubelet/pods/b758cbeb-2873-4a60-8781-ec74cd985a5f/volumes"
Nov 03 00:45:21.816509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.816493    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" path="/var/lib/kubelet/pods/e5c9eb1c-745a-403f-ae58-29e5b6c6d4db/volumes"
Nov 03 00:45:21.822315 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.822291    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"
Nov 03 00:45:21.822426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.822288    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt" event={"ID":"e4ae6ba2-9455-4a12-a5ca-b5064d128f34","Type":"ContainerDied","Data":"ba0ca98f9846cbba2511a5bce7d6adf13ed1708e3f08d2ae299ac927e92f0f07"}
Nov 03 00:45:21.822486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.822465    4550 scope.go:122] "RemoveContainer" containerID="db0f6dc07f982a4851609eac020caf12a1077d56eac53efd743adfa5db0e818f"
Nov 03 00:45:21.824764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.824739    4550 generic.go:356] "Generic (PLEG): container finished" podID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerID="528af1f5f213543894531ddb410d92fd37a990a546715e2e777d26e52009c5b8" exitCode=60
Nov 03 00:45:21.824896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.824873    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerDied","Data":"528af1f5f213543894531ddb410d92fd37a990a546715e2e777d26e52009c5b8"}
Nov 03 00:45:21.828986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.828843    4550 generic.go:356] "Generic (PLEG): container finished" podID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerID="f393bbf964cbc1313e99540b7e06906c72254bbd02800f007e1123f73225bf1e" exitCode=60
Nov 03 00:45:21.828986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.828930    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerDied","Data":"f393bbf964cbc1313e99540b7e06906c72254bbd02800f007e1123f73225bf1e"}
Nov 03 00:45:21.831549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.831016    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"
Nov 03 00:45:21.831549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.831027    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp" event={"ID":"6437d6ba-9677-43f2-93f9-c9b1762799e0","Type":"ContainerDied","Data":"951ec701ee52b061ea25e82e7149faca788ab895e98e3ef38170e98c37106a27"}
Nov 03 00:45:21.844165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.844115    4550 scope.go:122] "RemoveContainer" containerID="f060a4057d598658b00d9c4777f00a4dfde3020097e3bda546025c3b485ceb40"
Nov 03 00:45:21.846614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.846576    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerStarted","Data":"c49ad2ceaded6c2099b1c9a51fb7e3c8bcbd80c2415550da16e9418eedd8be04"}
Nov 03 00:45:21.849657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.849623    4550 generic.go:356] "Generic (PLEG): container finished" podID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerID="5179a168f8173d31fdb5948fcdd32b8b8e5dcf25d81db1d260a5a5d542e424ce" exitCode=60
Nov 03 00:45:21.849767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.849696    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerDied","Data":"5179a168f8173d31fdb5948fcdd32b8b8e5dcf25d81db1d260a5a5d542e424ce"}
Nov 03 00:45:21.850155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.850121    4550 scope.go:122] "RemoveContainer" containerID="5179a168f8173d31fdb5948fcdd32b8b8e5dcf25d81db1d260a5a5d542e424ce"
Nov 03 00:45:21.850589 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:21.850559    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn_odh-ai-gateway-infra(2044c984-2256-46c0-a939-bb5d39f2a9f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0"
Nov 03 00:45:21.854074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.854042    4550 generic.go:356] "Generic (PLEG): container finished" podID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerID="798391ae0513ec9b491d0766edfacc96264d80f4b5fee4d26fea5868a6c2ca27" exitCode=60
Nov 03 00:45:21.854225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.854125    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerDied","Data":"798391ae0513ec9b491d0766edfacc96264d80f4b5fee4d26fea5868a6c2ca27"}
Nov 03 00:45:21.855111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.854644    4550 scope.go:122] "RemoveContainer" containerID="798391ae0513ec9b491d0766edfacc96264d80f4b5fee4d26fea5868a6c2ca27"
Nov 03 00:45:21.855111 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:21.855054    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p_odh-ai-gateway-infra(ab7524e5-60fd-4d8a-8cac-75bd14006e93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93"
Nov 03 00:45:21.856744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.856723    4550 generic.go:356] "Generic (PLEG): container finished" podID="2a596167-f57e-4c78-8d4f-d817b460d090" containerID="ea58193d2ad6a17e15b3c68f5aa4d82c0dea4e14d68ba9e9e1b977121b033e96" exitCode=60
Nov 03 00:45:21.856856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.856810    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerDied","Data":"ea58193d2ad6a17e15b3c68f5aa4d82c0dea4e14d68ba9e9e1b977121b033e96"}
Nov 03 00:45:21.857534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.857176    4550 scope.go:122] "RemoveContainer" containerID="ea58193d2ad6a17e15b3c68f5aa4d82c0dea4e14d68ba9e9e1b977121b033e96"
Nov 03 00:45:21.857534 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:21.857479    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq_odh-ai-gateway-infra(2a596167-f57e-4c78-8d4f-d817b460d090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" podUID="2a596167-f57e-4c78-8d4f-d817b460d090"
Nov 03 00:45:21.882976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.882700    4550 scope.go:122] "RemoveContainer" containerID="9012ca3e4dfaa82901b1367f761232e96e79a9366bff27817fb7c4f2e251eac0"
Nov 03 00:45:21.939491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.937012    4550 scope.go:122] "RemoveContainer" containerID="1a9eb0a0cf485cef36c44bf54baef23c95d9b18fccd9798657dc22bdb04e5438"
Nov 03 00:45:21.981773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:21.981643    4550 scope.go:122] "RemoveContainer" containerID="dd0a7031772a43c58b07b5acf902c25be66432b2dc76a090e6ecd1f6ffafac5f"
Nov 03 00:45:22.059931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.059903    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:22.074713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.074643    4550 scope.go:122] "RemoveContainer" containerID="24e59a3dc6cdbd4846a1e11e2575bbf83810104a84a57b400c229db3ab10997b"
Nov 03 00:45:22.125702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.125294    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"]
Nov 03 00:45:22.139851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.139810    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894445-gbtxp"]
Nov 03 00:45:22.140113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.140094    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"]
Nov 03 00:45:22.162679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.149765    4550 scope.go:122] "RemoveContainer" containerID="4efbb7c33cbf5913e301ec64eaa7067da03e2e31c4a23fc8f8582749d926aea7"
Nov 03 00:45:22.185200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.184685    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:22.194684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.193981    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"]
Nov 03 00:45:22.199828 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:22.197730    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:22.199828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.198566    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l\" (UniqueName: \"kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l\") pod \"8d8abcb6-5936-438c-83a5-7050131bb1c5\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") "
Nov 03 00:45:22.199828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.198664    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca\") pod \"8d8abcb6-5936-438c-83a5-7050131bb1c5\" (UID: \"8d8abcb6-5936-438c-83a5-7050131bb1c5\") "
Nov 03 00:45:22.199828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.199598    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca" pod "8d8abcb6-5936-438c-83a5-7050131bb1c5" (UID: "8d8abcb6-5936-438c-83a5-7050131bb1c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:22.203766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.203740    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l" pod "8d8abcb6-5936-438c-83a5-7050131bb1c5" (UID: "8d8abcb6-5936-438c-83a5-7050131bb1c5"). InnerVolumeSpecName "kube-api-access-n4n8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:22.205424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.205401    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:22.207617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.207594    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:22.293337 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.293296    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"]
Nov 03 00:45:22.297039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.297009    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894445-k74pt"]
Nov 03 00:45:22.299749 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.299725    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca\") pod \"1bea83fd-c091-4b89-80e5-7728801df3f1\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") "
Nov 03 00:45:22.299912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.299787    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx\" (UniqueName: \"kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx\") pod \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") "
Nov 03 00:45:22.299912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.299820    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt\" (UniqueName: \"kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt\") pod \"1bea83fd-c091-4b89-80e5-7728801df3f1\" (UID: \"1bea83fd-c091-4b89-80e5-7728801df3f1\") "
Nov 03 00:45:22.299912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.299872    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca\") pod \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\" (UID: \"d9634e52-dcc3-4e14-b7ee-bfe7f207c870\") "
Nov 03 00:45:22.300074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.299945    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca\") pod \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") "
Nov 03 00:45:22.300074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.299998    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw\" (UniqueName: \"kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw\") pod \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\" (UID: \"d91f4d59-9ff5-4054-8e0d-7701138e5f5c\") "
Nov 03 00:45:22.300186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300139    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca" pod "1bea83fd-c091-4b89-80e5-7728801df3f1" (UID: "1bea83fd-c091-4b89-80e5-7728801df3f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:22.300358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300336    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca" pod "d91f4d59-9ff5-4054-8e0d-7701138e5f5c" (UID: "d91f4d59-9ff5-4054-8e0d-7701138e5f5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:22.300754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300731    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca" pod "d9634e52-dcc3-4e14-b7ee-bfe7f207c870" (UID: "d9634e52-dcc3-4e14-b7ee-bfe7f207c870"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:22.300958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300940    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bea83fd-c091-4b89-80e5-7728801df3f1-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.301053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300961    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.301053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300975    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4n8l\" (UniqueName: \"kubernetes.io/projected/8d8abcb6-5936-438c-83a5-7050131bb1c5-kube-api-access-n4n8l\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.301053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.300989    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d8abcb6-5936-438c-83a5-7050131bb1c5-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.302424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.302406    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx" pod "d9634e52-dcc3-4e14-b7ee-bfe7f207c870" (UID: "d9634e52-dcc3-4e14-b7ee-bfe7f207c870"). InnerVolumeSpecName "kube-api-access-m4bvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:22.302529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.302488    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw" pod "d91f4d59-9ff5-4054-8e0d-7701138e5f5c" (UID: "d91f4d59-9ff5-4054-8e0d-7701138e5f5c"). InnerVolumeSpecName "kube-api-access-7g2nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:22.302529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.302517    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt" pod "1bea83fd-c091-4b89-80e5-7728801df3f1" (UID: "1bea83fd-c091-4b89-80e5-7728801df3f1"). InnerVolumeSpecName "kube-api-access-2gvzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:22.303979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.303902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:22.344693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.344610    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"]
Nov 03 00:45:22.406093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.405991    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g2nw\" (UniqueName: \"kubernetes.io/projected/d91f4d59-9ff5-4054-8e0d-7701138e5f5c-kube-api-access-7g2nw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.406093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.406029    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bvx\" (UniqueName: \"kubernetes.io/projected/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-kube-api-access-m4bvx\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.406093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.406046    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gvzt\" (UniqueName: \"kubernetes.io/projected/1bea83fd-c091-4b89-80e5-7728801df3f1-kube-api-access-2gvzt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.406093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.406061    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9634e52-dcc3-4e14-b7ee-bfe7f207c870-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.879098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.879058    4550 generic.go:356] "Generic (PLEG): container finished" podID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerID="c49ad2ceaded6c2099b1c9a51fb7e3c8bcbd80c2415550da16e9418eedd8be04" exitCode=60
Nov 03 00:45:22.879284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.879102    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerDied","Data":"c49ad2ceaded6c2099b1c9a51fb7e3c8bcbd80c2415550da16e9418eedd8be04"}
Nov 03 00:45:22.879284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.879195    4550 scope.go:122] "RemoveContainer" containerID="84515673e7077a58873a8020cb729a3648d7290e933a637dc97b367f84152c1b"
Nov 03 00:45:22.879904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.879702    4550 scope.go:122] "RemoveContainer" containerID="c49ad2ceaded6c2099b1c9a51fb7e3c8bcbd80c2415550da16e9418eedd8be04"
Nov 03 00:45:22.880352 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:22.880315    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq_odh-ai-gateway-infra(a8014b63-70fd-4992-aa3d-7a6664cea4bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd"
Nov 03 00:45:22.892000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.891968    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w" event={"ID":"8d8abcb6-5936-438c-83a5-7050131bb1c5","Type":"ContainerDied","Data":"efe091c9f35bae8052a0108b46ac174e9a482d207116c290ff48cc76be125741"}
Nov 03 00:45:22.892138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.892075    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"
Nov 03 00:45:22.900178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.900153    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"
Nov 03 00:45:22.909829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.900334    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx" event={"ID":"1bea83fd-c091-4b89-80e5-7728801df3f1","Type":"ContainerDied","Data":"1525b5d2b8170837b98cce23ef5c2d46738a8ce1daec3ea5daadf1d46211ad92"}
Nov 03 00:45:22.913696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.912492    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q" event={"ID":"d91f4d59-9ff5-4054-8e0d-7701138e5f5c","Type":"ContainerDied","Data":"b3ea31d60790030e49764e2d0ae8da98371927a34aee5a8a0a417324eff3f4d2"}
Nov 03 00:45:22.913696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.912605    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"
Nov 03 00:45:22.913696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.912697    4550 scope.go:122] "RemoveContainer" containerID="528af1f5f213543894531ddb410d92fd37a990a546715e2e777d26e52009c5b8"
Nov 03 00:45:22.920641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.919331    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg" event={"ID":"d9634e52-dcc3-4e14-b7ee-bfe7f207c870","Type":"ContainerDied","Data":"90a218e90c50e8f5346aae90fad8c7ce653387d1c90c32973ce8c0791609a32f"}
Nov 03 00:45:22.920641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.919486    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"
Nov 03 00:45:22.974273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.974238    4550 scope.go:122] "RemoveContainer" containerID="f393bbf964cbc1313e99540b7e06906c72254bbd02800f007e1123f73225bf1e"
Nov 03 00:45:23.000479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:22.999047    4550 scope.go:122] "RemoveContainer" containerID="cbd777d33ead7efca6dd5a20cad28f90c4d7597bfb2f29f441839e07e7390598"
Nov 03 00:45:23.031583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.031551    4550 scope.go:122] "RemoveContainer" containerID="08d603ddc278c8173730963f9f6b49c75cff4b34feb9e4453e6119115252c4de"
Nov 03 00:45:23.045977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.045950    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"]
Nov 03 00:45:23.066564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.065950    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"]
Nov 03 00:45:23.081602 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.081532    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894445-cbr8w"]
Nov 03 00:45:23.135121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.135044    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:23.205248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.205215    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:23.206485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.206426    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"]
Nov 03 00:45:23.210185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.210162    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894445-7qw6q"]
Nov 03 00:45:23.214184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.214165    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:23.221425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.221403    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm\" (UniqueName: \"kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm\") pod \"2044c984-2256-46c0-a939-bb5d39f2a9f0\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") "
Nov 03 00:45:23.221541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.221487    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca\") pod \"2044c984-2256-46c0-a939-bb5d39f2a9f0\" (UID: \"2044c984-2256-46c0-a939-bb5d39f2a9f0\") "
Nov 03 00:45:23.221835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.221808    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca" pod "2044c984-2256-46c0-a939-bb5d39f2a9f0" (UID: "2044c984-2256-46c0-a939-bb5d39f2a9f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:23.222190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.222172    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044c984-2256-46c0-a939-bb5d39f2a9f0-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.223570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.223549    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm" pod "2044c984-2256-46c0-a939-bb5d39f2a9f0" (UID: "2044c984-2256-46c0-a939-bb5d39f2a9f0"). InnerVolumeSpecName "kube-api-access-vhxgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:23.241379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.241348    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"]
Nov 03 00:45:23.244941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.244913    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894445-27kdg"]
Nov 03 00:45:23.278381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.278347    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"]
Nov 03 00:45:23.295614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.295581    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894445-vjpxx"]
Nov 03 00:45:23.322758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.322720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca\") pod \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") "
Nov 03 00:45:23.322965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.322943    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca\") pod \"2a596167-f57e-4c78-8d4f-d817b460d090\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") "
Nov 03 00:45:23.323129 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.323109    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t\" (UniqueName: \"kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t\") pod \"2a596167-f57e-4c78-8d4f-d817b460d090\" (UID: \"2a596167-f57e-4c78-8d4f-d817b460d090\") "
Nov 03 00:45:23.323216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.323153    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq\" (UniqueName: \"kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq\") pod \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\" (UID: \"ab7524e5-60fd-4d8a-8cac-75bd14006e93\") "
Nov 03 00:45:23.323719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.323691    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca" pod "2a596167-f57e-4c78-8d4f-d817b460d090" (UID: "2a596167-f57e-4c78-8d4f-d817b460d090"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:23.323836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.323758    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxgm\" (UniqueName: \"kubernetes.io/projected/2044c984-2256-46c0-a939-bb5d39f2a9f0-kube-api-access-vhxgm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.324045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.324022    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca" pod "ab7524e5-60fd-4d8a-8cac-75bd14006e93" (UID: "ab7524e5-60fd-4d8a-8cac-75bd14006e93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:23.326610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.326581    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t" pod "2a596167-f57e-4c78-8d4f-d817b460d090" (UID: "2a596167-f57e-4c78-8d4f-d817b460d090"). InnerVolumeSpecName "kube-api-access-4m89t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:23.326754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.326734    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq" pod "ab7524e5-60fd-4d8a-8cac-75bd14006e93" (UID: "ab7524e5-60fd-4d8a-8cac-75bd14006e93"). InnerVolumeSpecName "kube-api-access-7wzlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:23.425342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.425302    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7524e5-60fd-4d8a-8cac-75bd14006e93-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.425342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.425345    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a596167-f57e-4c78-8d4f-d817b460d090-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.425580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.425355    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m89t\" (UniqueName: \"kubernetes.io/projected/2a596167-f57e-4c78-8d4f-d817b460d090-kube-api-access-4m89t\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.425580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.425366    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wzlq\" (UniqueName: \"kubernetes.io/projected/ab7524e5-60fd-4d8a-8cac-75bd14006e93-kube-api-access-7wzlq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.810830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.810747    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" path="/var/lib/kubelet/pods/1bea83fd-c091-4b89-80e5-7728801df3f1/volumes"
Nov 03 00:45:23.811286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.811268    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" path="/var/lib/kubelet/pods/6437d6ba-9677-43f2-93f9-c9b1762799e0/volumes"
Nov 03 00:45:23.811695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.811682    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" path="/var/lib/kubelet/pods/8d8abcb6-5936-438c-83a5-7050131bb1c5/volumes"
Nov 03 00:45:23.812352 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.812338    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" path="/var/lib/kubelet/pods/d91f4d59-9ff5-4054-8e0d-7701138e5f5c/volumes"
Nov 03 00:45:23.812741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.812729    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" path="/var/lib/kubelet/pods/d9634e52-dcc3-4e14-b7ee-bfe7f207c870/volumes"
Nov 03 00:45:23.813086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.813075    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" path="/var/lib/kubelet/pods/e4ae6ba2-9455-4a12-a5ca-b5064d128f34/volumes"
Nov 03 00:45:23.931138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.931103    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"
Nov 03 00:45:23.931138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.931113    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn" event={"ID":"2044c984-2256-46c0-a939-bb5d39f2a9f0","Type":"ContainerDied","Data":"8b74ffd8b295cf719f924a12f97b560cdba3b72149c4faf998547a30fe58c74c"}
Nov 03 00:45:23.931356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.931179    4550 scope.go:122] "RemoveContainer" containerID="5179a168f8173d31fdb5948fcdd32b8b8e5dcf25d81db1d260a5a5d542e424ce"
Nov 03 00:45:23.933138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.933111    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"
Nov 03 00:45:23.933585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.933562    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p" event={"ID":"ab7524e5-60fd-4d8a-8cac-75bd14006e93","Type":"ContainerDied","Data":"f58db1aec9a1f74f9c8af7ee9c228bba8a04e549d8cac79c445bf327fcb06ab5"}
Nov 03 00:45:23.940821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.940797    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"
Nov 03 00:45:23.941398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.941366    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq" event={"ID":"2a596167-f57e-4c78-8d4f-d817b460d090","Type":"ContainerDied","Data":"99eb072228cc770287e03ac8d8099a07270b1492fe169543716d91306b53260a"}
Nov 03 00:45:23.952882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.952858    4550 scope.go:122] "RemoveContainer" containerID="798391ae0513ec9b491d0766edfacc96264d80f4b5fee4d26fea5868a6c2ca27"
Nov 03 00:45:23.965032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:23.965012    4550 scope.go:122] "RemoveContainer" containerID="ea58193d2ad6a17e15b3c68f5aa4d82c0dea4e14d68ba9e9e1b977121b033e96"
Nov 03 00:45:24.018243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.018209    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"]
Nov 03 00:45:24.023935 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.023901    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894445-k76bq"]
Nov 03 00:45:24.050870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.050836    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"]
Nov 03 00:45:24.054709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.054681    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894445-hch4p"]
Nov 03 00:45:24.062517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.060498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:24.089310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.089275    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"]
Nov 03 00:45:24.095158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.095121    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a07a90-29894445-wtstn"]
Nov 03 00:45:24.139692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.139664    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:24.236415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.236381    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm\" (UniqueName: \"kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm\") pod \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") "
Nov 03 00:45:24.236415 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.236424    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca\") pod \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\" (UID: \"a8014b63-70fd-4992-aa3d-7a6664cea4bd\") "
Nov 03 00:45:24.236858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.236831    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca" pod "a8014b63-70fd-4992-aa3d-7a6664cea4bd" (UID: "a8014b63-70fd-4992-aa3d-7a6664cea4bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:24.237091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.237069    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8014b63-70fd-4992-aa3d-7a6664cea4bd-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.238614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.238595    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm" pod "a8014b63-70fd-4992-aa3d-7a6664cea4bd" (UID: "a8014b63-70fd-4992-aa3d-7a6664cea4bd"). InnerVolumeSpecName "kube-api-access-27hhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:24.338696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.338611    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hhm\" (UniqueName: \"kubernetes.io/projected/a8014b63-70fd-4992-aa3d-7a6664cea4bd-kube-api-access-27hhm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.724248 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:24.724213    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:45:24.956208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.956172    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq" event={"ID":"a8014b63-70fd-4992-aa3d-7a6664cea4bd","Type":"ContainerDied","Data":"ef539f4def2ab876e5d787039f6d79ff0bc860aa06e243a7c116deaaa03858d4"}
Nov 03 00:45:24.956398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.956243    4550 scope.go:122] "RemoveContainer" containerID="c49ad2ceaded6c2099b1c9a51fb7e3c8bcbd80c2415550da16e9418eedd8be04"
Nov 03 00:45:24.956398 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:24.956185    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"
Nov 03 00:45:25.011538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.011440    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"]
Nov 03 00:45:25.014947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.014920    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-909268-29894445-qbbqq"]
Nov 03 00:45:25.812267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.812227    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" path="/var/lib/kubelet/pods/2044c984-2256-46c0-a939-bb5d39f2a9f0/volumes"
Nov 03 00:45:25.812901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.812883    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" path="/var/lib/kubelet/pods/2a596167-f57e-4c78-8d4f-d817b460d090/volumes"
Nov 03 00:45:25.813311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.813292    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" path="/var/lib/kubelet/pods/a8014b63-70fd-4992-aa3d-7a6664cea4bd/volumes"
Nov 03 00:45:25.813999 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.813979    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" path="/var/lib/kubelet/pods/ab7524e5-60fd-4d8a-8cac-75bd14006e93/volumes"
Nov 03 00:45:25.826003 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:25.825968    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:27.405848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:27.405814    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:29.057717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:29.057679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:30.730730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:30.730695    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:32.276579 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:32.276536    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:32.480011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:32.479975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:32.578299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:32.578205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:32.679600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:32.679555    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:33.222441 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:33.222378    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:34.413227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:34.413197    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:35.115763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:35.115696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:35.596942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:35.596907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:35.713491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:35.712928    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:35.910526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:35.905201    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:36.082994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.076166    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"]
Nov 03 00:45:36.111454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.103953    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3089eaa7-f4f8c"]
Nov 03 00:45:36.482311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.481201    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:36.785527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.785396    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"]
Nov 03 00:45:36.787065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787034    4550 cpu_manager.go:414] "containerMap: removing container" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:45:36.787065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787063    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787074    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787082    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787099    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787106    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787115    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787124    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787139    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787147    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787187    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787195    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787208    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787214    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787223    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787230    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787239    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787246    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787259    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787266    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:45:36.787270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787278    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787285    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787295    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787302    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787315    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787321    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787334    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787342    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787356    4550 cpu_manager.go:414] "containerMap: removing container" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787362    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787376    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787382    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787397    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787404    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787414    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787423    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787454    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787461    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787469    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787476    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787502    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787510    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787522    4550 cpu_manager.go:414] "containerMap: removing container" podUID="18de8fc3-d694-4f27-935d-bef519be6005" containerName="payload-processing"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787529    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18de8fc3-d694-4f27-935d-bef519be6005" containerName="payload-processing"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787542    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787549    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787559    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787566    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787577    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787584    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787594    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787601    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787620    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787627    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787639    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787646    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787657    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787664    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787673    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787680    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787689    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787696    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787711    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787718    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787733    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787739    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787748    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787755    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787765    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787772    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787814    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787822    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787833    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787839    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787852    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787859    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787870    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787876    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787905    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.787913    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788067    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788079    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788088    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788099    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788109    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788118    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788125    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788133    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788141    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788150    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788159    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788168    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788176    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788186    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788194    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788201    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788209    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="18de8fc3-d694-4f27-935d-bef519be6005" containerName="payload-processing"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788218    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788226    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788233    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788242    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788250    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8014b63-70fd-4992-aa3d-7a6664cea4bd" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788261    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788268    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788275    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788283    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788291    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788298    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788308    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788316    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788324    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788332    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:45:36.788278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788340    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:45:36.793090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788350    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:45:36.793090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788359    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:45:36.793090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788368    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:45:36.793090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.788375    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:45:36.794537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.794512    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"]
Nov 03 00:45:36.794671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.794618    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:36.828322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828276    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-536b490d5ab1\""
Nov 03 00:45:36.828614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828344    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"openshift-service-ca.crt\""
Nov 03 00:45:36.828614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828398    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-e2e--2e23f326-dockercfg-njdlx\""
Nov 03 00:45:36.828614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828421    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-aa752df7c9fc\""
Nov 03 00:45:36.828614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828536    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"kube-root-ca.crt\""
Nov 03 00:45:36.828614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828540    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdacdb\""
Nov 03 00:45:36.828614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828605    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-2274efc91ed5\""
Nov 03 00:45:36.828981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.828638    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"routing-overlay\""
Nov 03 00:45:36.883206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.883107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:36.936486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.934935    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:36.936486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.935002    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:36.936486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.935179    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8dp\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:36.936486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.935282    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:36.994154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:36.994115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:37.037053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.036962    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.037053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.037016    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.037259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.037107    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8dp\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.037259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.037187    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.038154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.038122    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.038154 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.038148    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.040079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.040054    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.071455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.068416    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8dp\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.121454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.118693    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:37.799502 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:45:37.799472    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9239ef4b_e590_4b45_a949_6fbe596766d4.slice/crio-0c8ead4fa2980cd70c852879ff64b4c4b7407a46bc21d7d2614b5c8fb9aac826 WatchSource:0}: Error finding container 0c8ead4fa2980cd70c852879ff64b4c4b7407a46bc21d7d2614b5c8fb9aac826: Status 404 returned error can't find the container with id 0c8ead4fa2980cd70c852879ff64b4c4b7407a46bc21d7d2614b5c8fb9aac826
Nov 03 00:45:37.831956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.826398    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29cf92c-3f62-43ba-88ba-f3fd4c553860" path="/var/lib/kubelet/pods/f29cf92c-3f62-43ba-88ba-f3fd4c553860/volumes"
Nov 03 00:45:37.831956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:37.826893    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"]
Nov 03 00:45:38.295417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:38.295378    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj" event={"ID":"9239ef4b-e590-4b45-a949-6fbe596766d4","Type":"ContainerStarted","Data":"7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea"}
Nov 03 00:45:38.295671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:38.295469    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj" event={"ID":"9239ef4b-e590-4b45-a949-6fbe596766d4","Type":"ContainerStarted","Data":"0c8ead4fa2980cd70c852879ff64b4c4b7407a46bc21d7d2614b5c8fb9aac826"}
Nov 03 00:45:38.295671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:38.295526    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:38.337930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:38.337872    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj" podStartSLOduration=2.3378572269999998 podStartE2EDuration="2.337857227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:38.334316772 +0000 UTC m=+2420.655853571" watchObservedRunningTime="2026-11-03 00:45:38.337857227 +0000 UTC m=+2420.659394025"
Nov 03 00:45:38.506072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:38.506031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:39.446447 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:39.446398    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:39.446447 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:39.446427    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:40.111747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:40.111715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:41.712584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:41.712539    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:45:41.713232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:41.713202    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:45:41.816051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:41.816015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:42.291348 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:42.291303    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:43.601875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:43.601840    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:44.279092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.279055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:44.361089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.359608    4550 generic.go:356] "Generic (PLEG): container finished" podID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerID="1ec4bdb682700f9298dbb93ad66bab7c02add3faeb25332c8cd8cb927ff5338b" exitCode=0
Nov 03 00:45:44.361089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.359853    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" event={"ID":"05672915-3fd5-4fdd-af1b-36ad57cff170","Type":"ContainerDied","Data":"1ec4bdb682700f9298dbb93ad66bab7c02add3faeb25332c8cd8cb927ff5338b"}
Nov 03 00:45:44.361089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.360461    4550 scope.go:122] "RemoveContainer" containerID="1ec4bdb682700f9298dbb93ad66bab7c02add3faeb25332c8cd8cb927ff5338b"
Nov 03 00:45:44.368341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.368284    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:45:44.368510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.368400    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:45:44.368510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:44.368467    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:45:45.233725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.233690    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:45.372177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.372139    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" event={"ID":"05672915-3fd5-4fdd-af1b-36ad57cff170","Type":"ContainerStarted","Data":"449a6396e38dfb9408d3393ce2da68f24a5e217e82b691ae0f06201a0e347c0b"}
Nov 03 00:45:45.401563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.400982    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:45:45.401563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.401037    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 00:45:45.401563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.401319    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:45:45.492623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.492523    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:45.596466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.596414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:45.788277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:45.788039    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:46.034926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.034885    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"]
Nov 03 00:45:46.044605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.044528    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57bd807b-t52pk"]
Nov 03 00:45:46.117231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.117196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:46.295647 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.295533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:46.398473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.396757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:46.399939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.399909    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:45:46.400146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:46.400126    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 00:45:47.067457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:47.063629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:47.189855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:47.189818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:47.311452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:47.305950    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:47.402593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:47.402553    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:45:47.402907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:47.402879    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 00:45:47.810796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:47.810713    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e01db93-cbac-48a5-a6c3-02671b416605" path="/var/lib/kubelet/pods/3e01db93-cbac-48a5-a6c3-02671b416605/volumes"
Nov 03 00:45:48.288450 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:48.288019    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:48.403468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:48.399828    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:45:48.403468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:48.399883    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 00:45:48.934817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:48.934783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:49.306314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:49.306217    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:45:49.399937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:49.399900    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:45:49.400240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:49.400210    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 00:45:50.400469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:50.400105    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Nov 03 00:45:50.400469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:50.400154    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Nov 03 00:45:50.618733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:50.618697    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:51.406603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:51.406568    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:45:51.407147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:51.407127    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:45:51.407809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:51.407792    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 00:45:52.307560 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:52.307521    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:45:52.371965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:52.371926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:54.471101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:54.471064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:54.725761 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:45:54.725676    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:45:56.229100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:56.229065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:57.937030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:57.936980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:59.377677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:59.377641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:59.491612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:59.491564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:59.751472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:59.751422    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:45:59.988053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:45:59.988018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:01.348581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:01.348540    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:02.398909 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:46:02.398872    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:46:03.037152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:03.037109    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:46:03.037357 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:03.037183    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:46:03.038397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:03.038366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:03.203615 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:46:03.203567    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:46:04.773761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:04.773726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:05.580032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:05.579992    4550 generic.go:356] "Generic (PLEG): container finished" podID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerID="f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410" exitCode=0
Nov 03 00:46:05.580225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:05.580025    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" event={"ID":"96b3eb4a-d717-44b7-b0bc-18f8997e97dd","Type":"ContainerDied","Data":"f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"}
Nov 03 00:46:05.580660 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:05.580636    4550 scope.go:122] "RemoveContainer" containerID="f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"
Nov 03 00:46:05.582727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:05.582694    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:46:05.582849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:05.582783    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:46:05.582849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:05.582832    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:46:06.278454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:06.271323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:06.374456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:06.374050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:06.590247 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:06.590136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:06.596388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:06.596352    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" event={"ID":"96b3eb4a-d717-44b7-b0bc-18f8997e97dd","Type":"ContainerStarted","Data":"ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67"}
Nov 03 00:46:06.713779 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:06.713731    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:07.062941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:07.062896    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:46:07.063377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:07.063286    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 00:46:07.063575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:07.063553    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 00:46:08.081456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:08.064888    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 00:46:08.081456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:08.064950    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 00:46:08.318752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:08.318716    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:09.064452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:09.060887    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 00:46:09.064452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:09.060951    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 00:46:09.981565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:09.981514    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:10.076451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:10.066866    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 00:46:10.076451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:10.066931    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 00:46:11.068465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:11.068080    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Nov 03 00:46:11.068465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:11.068141    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Nov 03 00:46:11.808078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:11.808043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:12.066846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:12.066766    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:46:12.067177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:12.067154    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:46:12.067939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:12.067919    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 00:46:12.145140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:12.145109    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:12.485587 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:46:12.485541    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc1e19a_e169_4974_a452_3a9446f4b3ac.slice/crio-conmon-e186a94f9334e4e5fb3b4434a299c001f341e4fa05e4a63b0cfa247830748059.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:46:12.503744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:12.503705    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:12.623598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:12.623555    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:13.012301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:13.012258    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:13.200916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:13.200876    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:13.887096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:13.887048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:14.019081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:14.019043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:15.633830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:15.633793    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:17.332599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:17.332557    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:18.995888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:18.995849    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:19.540655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:19.540619    4550 scope.go:122] "RemoveContainer" containerID="4e6a89081bdd190e30b7e17082dd28afa24d6fa5d1b138ca1bdce4ef45d3da27"
Nov 03 00:46:19.556797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:19.556766    4550 scope.go:122] "RemoveContainer" containerID="3eda3dee47383073b4d0da7f0c475f91f0ab491212c7b39d4ab73ea8927a128e"
Nov 03 00:46:20.596955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:20.596922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:22.336832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:22.336797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:24.025668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:24.025629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:24.728609 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:46:24.728573    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:46:24.757775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:24.757734    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:25.652811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:25.652774    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:26.017506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:26.017398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:26.130387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:26.130351    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:26.451797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:26.451759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:26.625758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:26.625721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:26.786424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:26.786331    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:26.887209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:26.887174    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:27.499638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:27.499595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:27.618191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:27.618156    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:27.716475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:27.716413    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:28.716972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:28.716928    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:29.536212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:29.536120    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:31.452835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:31.452796    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:32.530505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:32.530460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:32.631677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:32.631643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:33.397262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:33.397213    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:34.029751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:34.029719    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"]
Nov 03 00:46:34.042205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:34.042176    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab35fe9d-8mbpb"]
Nov 03 00:46:35.029638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:35.029605    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"]
Nov 03 00:46:35.042395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:35.042362    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e51c09a6-96vdk"]
Nov 03 00:46:35.225161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:35.225121    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:35.810347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:35.810312    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04155849-815d-47af-81d2-e8bc8683426d" path="/var/lib/kubelet/pods/04155849-815d-47af-81d2-e8bc8683426d/volumes"
Nov 03 00:46:35.810872 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:35.810856    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d95e483-63cd-4e6c-a0f1-085254a10250" path="/var/lib/kubelet/pods/6d95e483-63cd-4e6c-a0f1-085254a10250/volumes"
Nov 03 00:46:36.746870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:36.746831    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:36.847099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:36.847063    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:38.618781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:38.618746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:40.347788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:40.347745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:42.119269 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:42.119229    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:42.664504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:42.664465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:43.723705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:43.723670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:45.029510 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:45.029474    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"]
Nov 03 00:46:45.042552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:45.042519    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a5ca5d9-twmjx"]
Nov 03 00:46:45.371212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:45.371117    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:45.810358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:45.810316    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774409d7-ec27-43c9-812c-4687a4632105" path="/var/lib/kubelet/pods/774409d7-ec27-43c9-812c-4687a4632105/volumes"
Nov 03 00:46:46.028607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:46.028569    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"]
Nov 03 00:46:46.041676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:46.041645    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94c0b7d-r9psc"]
Nov 03 00:46:47.215848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:47.215816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:47.809728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:47.809693    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc36ba9-924b-4113-9ca7-4a8ed2db6458" path="/var/lib/kubelet/pods/bfc36ba9-924b-4113-9ca7-4a8ed2db6458/volumes"
Nov 03 00:46:47.855597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:47.855561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:48.351840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:48.351804    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:48.758395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:48.758357    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:49.037057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.036947    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:49.352680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.352571    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"]
Nov 03 00:46:49.652535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.652498    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"]
Nov 03 00:46:49.653387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653368    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:46:49.653451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653391    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:46:49.653451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653425    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653449    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653462    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653472    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7524e5-60fd-4d8a-8cac-75bd14006e93" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653488    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653494    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653509    4550 cpu_manager.go:414] "containerMap: removing container" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:46:49.653523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653516    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653531    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653538    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653562    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653568    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b758cbeb-2873-4a60-8781-ec74cd985a5f" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653593    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653599    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9a0a98-a505-4165-be89-7b45f49226e8" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653609    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:46:49.653732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653615    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653785    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a596167-f57e-4c78-8d4f-d817b460d090" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653800    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d8abcb6-5936-438c-83a5-7050131bb1c5" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653810    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5201b966-0adb-46a4-bfbb-79edcd61db78" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653819    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4ae6ba2-9455-4a12-a5ca-b5064d128f34" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653829    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91f4d59-9ff5-4054-8e0d-7701138e5f5c" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653838    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c9eb1c-745a-403f-ae58-29e5b6c6d4db" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653850    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6437d6ba-9677-43f2-93f9-c9b1762799e0" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653860    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2044c984-2256-46c0-a939-bb5d39f2a9f0" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653870    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9634e52-dcc3-4e14-b7ee-bfe7f207c870" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653882    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="56666705-dfb5-4876-b27b-22ee10e32c11" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653889    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bea83fd-c091-4b89-80e5-7728801df3f1" containerName="cleanup"
Nov 03 00:46:49.653972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.653897    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc1e19a-e169-4974-a452-3a9446f4b3ac" containerName="cleanup"
Nov 03 00:46:49.657949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.657930    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.674963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.674934    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fdacdb-openshift-default-dockercfg-2krd9\""
Nov 03 00:46:49.693093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.693063    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"]
Nov 03 00:46:49.725890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.725855    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.725907    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k6rh\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726018    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726109    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726263    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726633    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726789    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.726850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.726841    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.827707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827674    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.827932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827770    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.827932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827815    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.827932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827875    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.827932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827907    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k6rh\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827934    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.827971    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828040    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828082    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828292    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828503    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828532    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828606    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.828763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.828646    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.830484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.830462    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.830586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.830505    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.848799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.848770    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k6rh\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.848938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.848829    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:49.972743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:49.972657    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:50.287572 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:46:50.287537    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff25e55a_ca15_4609_afcc_656560b9bc51.slice/crio-7a823876d79114ddc8f386d515603fe9e7d5ae95c4cf071c91e86404c4c505da WatchSource:0}: Error finding container 7a823876d79114ddc8f386d515603fe9e7d5ae95c4cf071c91e86404c4c505da: Status 404 returned error can't find the container with id 7a823876d79114ddc8f386d515603fe9e7d5ae95c4cf071c91e86404c4c505da
Nov 03 00:46:50.288955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.288929    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"]
Nov 03 00:46:50.290632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.290602    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:46:50.290723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.290665    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:46:50.290723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.290696    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:46:50.631655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.631568    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:50.941058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.941015    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" event={"ID":"ff25e55a-ca15-4609-afcc-656560b9bc51","Type":"ContainerStarted","Data":"2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6"}
Nov 03 00:46:50.941259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.941069    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" event={"ID":"ff25e55a-ca15-4609-afcc-656560b9bc51","Type":"ContainerStarted","Data":"7a823876d79114ddc8f386d515603fe9e7d5ae95c4cf071c91e86404c4c505da"}
Nov 03 00:46:50.978524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:50.978490    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:51.021617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:51.021563    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podStartSLOduration=2.021549092 podStartE2EDuration="2.021549092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:51.005920316 +0000 UTC m=+2493.327457115" watchObservedRunningTime="2026-11-03 00:46:51.021549092 +0000 UTC m=+2493.343085889"
Nov 03 00:46:51.983628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:51.983571    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 00:46:51.984173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:51.983643    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 00:46:52.393499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:52.393375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:52.980915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:52.980858    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 00:46:52.981144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:52.980940    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 00:46:53.983981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:53.983943    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 00:46:53.987572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:53.987533    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 00:46:54.188764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.188726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:54.353859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.353769    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj" podUID="9239ef4b-e590-4b45-a949-6fbe596766d4" containerName="payload-processing" containerID="cri-o://7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea" gracePeriod=25
Nov 03 00:46:54.495082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.495020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:54.621498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.621347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:54.730262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.730233    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:46:54.737984 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:46:54.737949    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:46:54.742901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.742876    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:54.824925 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.824891    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials\") pod \"9239ef4b-e590-4b45-a949-6fbe596766d4\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") "
Nov 03 00:46:54.825075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.824996    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp\") pod \"9239ef4b-e590-4b45-a949-6fbe596766d4\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") "
Nov 03 00:46:54.825075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.825028    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume\") pod \"9239ef4b-e590-4b45-a949-6fbe596766d4\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") "
Nov 03 00:46:54.825075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.825068    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume\") pod \"9239ef4b-e590-4b45-a949-6fbe596766d4\" (UID: \"9239ef4b-e590-4b45-a949-6fbe596766d4\") "
Nov 03 00:46:54.825760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.825503    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume" pod "9239ef4b-e590-4b45-a949-6fbe596766d4" (UID: "9239ef4b-e590-4b45-a949-6fbe596766d4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:54.825897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.825875    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume" pod "9239ef4b-e590-4b45-a949-6fbe596766d4" (UID: "9239ef4b-e590-4b45-a949-6fbe596766d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:54.826171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.826150    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:46:54.826264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.826178    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9239ef4b-e590-4b45-a949-6fbe596766d4-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:46:54.827822 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.827792    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials" pod "9239ef4b-e590-4b45-a949-6fbe596766d4" (UID: "9239ef4b-e590-4b45-a949-6fbe596766d4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:54.828744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.828713    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp" pod "9239ef4b-e590-4b45-a949-6fbe596766d4" (UID: "9239ef4b-e590-4b45-a949-6fbe596766d4"). InnerVolumeSpecName "kube-api-access-6x8dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:54.927888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.927857    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:46:54.927888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.927894    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8dp\" (UniqueName: \"kubernetes.io/projected/9239ef4b-e590-4b45-a949-6fbe596766d4-kube-api-access-6x8dp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:46:54.990490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.987856    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 00:46:54.990490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.987916    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 00:46:55.001486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.998642    4550 generic.go:356] "Generic (PLEG): container finished" podID="9239ef4b-e590-4b45-a949-6fbe596766d4" containerID="7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea" exitCode=0
Nov 03 00:46:55.001486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.998708    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"
Nov 03 00:46:55.001486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.998720    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj" event={"ID":"9239ef4b-e590-4b45-a949-6fbe596766d4","Type":"ContainerDied","Data":"7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea"}
Nov 03 00:46:55.001486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.998791    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj" event={"ID":"9239ef4b-e590-4b45-a949-6fbe596766d4","Type":"ContainerDied","Data":"0c8ead4fa2980cd70c852879ff64b4c4b7407a46bc21d7d2614b5c8fb9aac826"}
Nov 03 00:46:55.001486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:54.998819    4550 scope.go:122] "RemoveContainer" containerID="7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea"
Nov 03 00:46:55.014028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.013582    4550 scope.go:122] "RemoveContainer" containerID="7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea"
Nov 03 00:46:55.014028 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:46:55.013853    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea\": container with ID starting with 7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea not found: ID does not exist" containerID="7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea"
Nov 03 00:46:55.014028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.013878    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea"} err="failed to get container status \"7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea\": rpc error: code = NotFound desc = could not find container \"7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea\": container with ID starting with 7751ad1d48b5a7234f420b6b9d51bd6dc8f350210dc1a0e20db98f0b6a20acea not found: ID does not exist"
Nov 03 00:46:55.073467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.069133    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"]
Nov 03 00:46:55.092655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.088996    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5cb74tmhj"]
Nov 03 00:46:55.809397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.809365    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9239ef4b-e590-4b45-a949-6fbe596766d4" path="/var/lib/kubelet/pods/9239ef4b-e590-4b45-a949-6fbe596766d4/volumes"
Nov 03 00:46:55.976074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.976040    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 00:46:55.976404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:55.976379    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 00:46:56.022775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:56.022738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:56.979115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:56.979081    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:56.979476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:56.979458    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:56.980189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:56.980165    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 00:46:57.814752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:57.814718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:46:59.509700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:46:59.509669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:00.149179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:00.149125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:00.250034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:00.249990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:01.111350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:01.111312    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:01.441499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:01.441423    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:03.143446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:03.143391    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:04.972050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:04.972006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:06.718897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:06.718856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:07.162598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.162567    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:07.264459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.264401    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:07.352110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.352075    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"]
Nov 03 00:47:07.352753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.352734    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9239ef4b-e590-4b45-a949-6fbe596766d4" containerName="payload-processing"
Nov 03 00:47:07.352834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.352755    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9239ef4b-e590-4b45-a949-6fbe596766d4" containerName="payload-processing"
Nov 03 00:47:07.352884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.352866    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9239ef4b-e590-4b45-a949-6fbe596766d4" containerName="payload-processing"
Nov 03 00:47:07.358084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.358057    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.371741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.371707    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8znv\" (UniqueName: \"kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.371899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.371866    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.371965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.371937    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.377685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.377651    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"]
Nov 03 00:47:07.388489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.386658    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:47:07.388489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.386670    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:47:07.473912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.473818    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8znv\" (UniqueName: \"kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.474095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.473986    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.474095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.474052    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.474940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.474914    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.476562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.476536    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.494482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.494421    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8znv\" (UniqueName: \"kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv\") pod \"maas-api-revoke-sub-d0ccfbee-wbx9j\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.679782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.679750    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:07.984303 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:07.984263    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"]
Nov 03 00:47:08.093815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:08.093766    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j" event={"ID":"2be9ed17-fa43-41f9-a174-f709b34092ba","Type":"ContainerStarted","Data":"3143d15f9d1a03ba2489aa88e7350aaad7e1d026ea5aca143281877967c3ecf9"}
Nov 03 00:47:08.131800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:08.131767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:08.232069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:08.232033    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:08.691346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:08.691273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:09.104830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:09.104788    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j" event={"ID":"2be9ed17-fa43-41f9-a174-f709b34092ba","Type":"ContainerStarted","Data":"ee6531e2da03b5f07ab30271838c37c3bf4cb0b6b2952d0d4ff0602641086ec6"}
Nov 03 00:47:09.143853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:09.143802    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j" podStartSLOduration=2.143789275 podStartE2EDuration="2.143789275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:09.142944446 +0000 UTC m=+2511.464481263" watchObservedRunningTime="2026-11-03 00:47:09.143789275 +0000 UTC m=+2511.465326074"
Nov 03 00:47:10.113212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:10.113181    4550 generic.go:356] "Generic (PLEG): container finished" podID="2be9ed17-fa43-41f9-a174-f709b34092ba" containerID="ee6531e2da03b5f07ab30271838c37c3bf4cb0b6b2952d0d4ff0602641086ec6" exitCode=0
Nov 03 00:47:10.113601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:10.113262    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j" event={"ID":"2be9ed17-fa43-41f9-a174-f709b34092ba","Type":"ContainerDied","Data":"ee6531e2da03b5f07ab30271838c37c3bf4cb0b6b2952d0d4ff0602641086ec6"}
Nov 03 00:47:10.418978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:10.418945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:10.921473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:10.921414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:11.486702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.486668    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:11.597933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.597896    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca\") pod \"2be9ed17-fa43-41f9-a174-f709b34092ba\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") "
Nov 03 00:47:11.598149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.597960    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca\") pod \"2be9ed17-fa43-41f9-a174-f709b34092ba\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") "
Nov 03 00:47:11.598219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.598199    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv\" (UniqueName: \"kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv\") pod \"2be9ed17-fa43-41f9-a174-f709b34092ba\" (UID: \"2be9ed17-fa43-41f9-a174-f709b34092ba\") "
Nov 03 00:47:11.598369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.598347    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca" pod "2be9ed17-fa43-41f9-a174-f709b34092ba" (UID: "2be9ed17-fa43-41f9-a174-f709b34092ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:11.598800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.598777    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:11.600111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.600088    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca" pod "2be9ed17-fa43-41f9-a174-f709b34092ba" (UID: "2be9ed17-fa43-41f9-a174-f709b34092ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:11.600279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.600260    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv" pod "2be9ed17-fa43-41f9-a174-f709b34092ba" (UID: "2be9ed17-fa43-41f9-a174-f709b34092ba"). InnerVolumeSpecName "kube-api-access-h8znv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:11.700103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.700066    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8znv\" (UniqueName: \"kubernetes.io/projected/2be9ed17-fa43-41f9-a174-f709b34092ba-kube-api-access-h8znv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:11.700103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:11.700102    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be9ed17-fa43-41f9-a174-f709b34092ba-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:12.134471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:12.134363    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j" event={"ID":"2be9ed17-fa43-41f9-a174-f709b34092ba","Type":"ContainerDied","Data":"3143d15f9d1a03ba2489aa88e7350aaad7e1d026ea5aca143281877967c3ecf9"}
Nov 03 00:47:12.134471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:12.134395    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"
Nov 03 00:47:12.134471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:12.134414    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3143d15f9d1a03ba2489aa88e7350aaad7e1d026ea5aca143281877967c3ecf9"
Nov 03 00:47:12.220039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:12.220007    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:12.333449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:12.333388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:14.017755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:14.017723    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:15.662067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:15.662030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:15.757565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:15.757531    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:17.532220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:17.532166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:18.234816 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.234777    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"]
Nov 03 00:47:18.246936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.244794    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2be9ed17-fa43-41f9-a174-f709b34092ba" containerName="revoke-keys"
Nov 03 00:47:18.246936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.244826    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be9ed17-fa43-41f9-a174-f709b34092ba" containerName="revoke-keys"
Nov 03 00:47:18.246936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.245036    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be9ed17-fa43-41f9-a174-f709b34092ba" containerName="revoke-keys"
Nov 03 00:47:18.250634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.250607    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.267202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.267174    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"]
Nov 03 00:47:18.272096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.272071    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:47:18.272604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.272583    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:47:18.399629 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.399593    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"]
Nov 03 00:47:18.403829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.403810    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.413030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.413002    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.413223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.413201    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npf8m\" (UniqueName: \"kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.413271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.413247    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.421267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.421238    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"]
Nov 03 00:47:18.514463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.514351    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npf8m\" (UniqueName: \"kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.514463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.514409    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.514681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.514474    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.514681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.514497    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.514764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.514734    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.514801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.514763    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl74\" (UniqueName: \"kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.515395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.515377    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.517119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.517097    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.534950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.534915    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npf8m\" (UniqueName: \"kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m\") pod \"maas-api-revoke-sub-e5343b68-pvtn6\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.580642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.580607    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:18.616730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.616690    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl74\" (UniqueName: \"kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.617010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.616987    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.617095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.617047    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.617791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.617768    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.619636 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.619612    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.641116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.641087    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl74\" (UniqueName: \"kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74\") pod \"maas-api-revoke-sub-cc5fce51-2p54p\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.718818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.717911    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:18.886342 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:47:18.886286    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09887ad0_66c2_48b5_9b75_34b6a4f4c313.slice/crio-5fa833bcd369378a1f7637fd038ebdc7e6bf878e8de6c9dfae4c02e3672121f8 WatchSource:0}: Error finding container 5fa833bcd369378a1f7637fd038ebdc7e6bf878e8de6c9dfae4c02e3672121f8: Status 404 returned error can't find the container with id 5fa833bcd369378a1f7637fd038ebdc7e6bf878e8de6c9dfae4c02e3672121f8
Nov 03 00:47:18.889499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:18.889461    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"]
Nov 03 00:47:19.024907 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:47:19.024874    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc567f22_d0d7_421f_8a05_5941f128ac44.slice/crio-0a7769ba280ce3fa41152688c31d8b9f5ca6ae2377592d2ca5abfbe466547806 WatchSource:0}: Error finding container 0a7769ba280ce3fa41152688c31d8b9f5ca6ae2377592d2ca5abfbe466547806: Status 404 returned error can't find the container with id 0a7769ba280ce3fa41152688c31d8b9f5ca6ae2377592d2ca5abfbe466547806
Nov 03 00:47:19.032198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.032174    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"]
Nov 03 00:47:19.188539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.188498    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6" event={"ID":"09887ad0-66c2-48b5-9b75-34b6a4f4c313","Type":"ContainerStarted","Data":"04fadcc566ad963ef60a1872bdf8db3a1269763641b54d084b8850861acf8759"}
Nov 03 00:47:19.188803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.188568    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6" event={"ID":"09887ad0-66c2-48b5-9b75-34b6a4f4c313","Type":"ContainerStarted","Data":"5fa833bcd369378a1f7637fd038ebdc7e6bf878e8de6c9dfae4c02e3672121f8"}
Nov 03 00:47:19.190488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.190458    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p" event={"ID":"fc567f22-d0d7-421f-8a05-5941f128ac44","Type":"ContainerStarted","Data":"0a7769ba280ce3fa41152688c31d8b9f5ca6ae2377592d2ca5abfbe466547806"}
Nov 03 00:47:19.233623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.233562    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6" podStartSLOduration=1.233548854 podStartE2EDuration="1.233548854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:19.229750927 +0000 UTC m=+2521.551287736" watchObservedRunningTime="2026-11-03 00:47:19.233548854 +0000 UTC m=+2521.555085653"
Nov 03 00:47:19.247751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.247718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:19.673706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.673672    4550 scope.go:122] "RemoveContainer" containerID="033a37b02c905143bd867afb92d8ca254f056ef0c108b6e7e0f62b4e87bfc69a"
Nov 03 00:47:19.684758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:19.684732    4550 scope.go:122] "RemoveContainer" containerID="1efb07aa44d8eee7e3cf02c65cd2a640d458884ebc686252b4c484388c9f1b96"
Nov 03 00:47:20.198675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:20.198639    4550 generic.go:356] "Generic (PLEG): container finished" podID="09887ad0-66c2-48b5-9b75-34b6a4f4c313" containerID="04fadcc566ad963ef60a1872bdf8db3a1269763641b54d084b8850861acf8759" exitCode=0
Nov 03 00:47:20.198899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:20.198730    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6" event={"ID":"09887ad0-66c2-48b5-9b75-34b6a4f4c313","Type":"ContainerDied","Data":"04fadcc566ad963ef60a1872bdf8db3a1269763641b54d084b8850861acf8759"}
Nov 03 00:47:20.200308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:20.200283    4550 generic.go:356] "Generic (PLEG): container finished" podID="fc567f22-d0d7-421f-8a05-5941f128ac44" containerID="5b7e7a6a251c1ff52fb04a723cb451f908c0bc7a22c6d9bffb23ecc600b6aa8c" exitCode=0
Nov 03 00:47:20.200409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:20.200349    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p" event={"ID":"fc567f22-d0d7-421f-8a05-5941f128ac44","Type":"ContainerDied","Data":"5b7e7a6a251c1ff52fb04a723cb451f908c0bc7a22c6d9bffb23ecc600b6aa8c"}
Nov 03 00:47:20.767475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:20.767411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:20.922869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:20.922808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:21.391495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.391425    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:21.425049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.425020    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:21.561175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.561078    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74\" (UniqueName: \"kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74\") pod \"fc567f22-d0d7-421f-8a05-5941f128ac44\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") "
Nov 03 00:47:21.561175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.561118    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca\") pod \"fc567f22-d0d7-421f-8a05-5941f128ac44\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") "
Nov 03 00:47:21.561175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.561162    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca\") pod \"fc567f22-d0d7-421f-8a05-5941f128ac44\" (UID: \"fc567f22-d0d7-421f-8a05-5941f128ac44\") "
Nov 03 00:47:21.562086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.561737    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca\") pod \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") "
Nov 03 00:47:21.562086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.562016    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca" pod "fc567f22-d0d7-421f-8a05-5941f128ac44" (UID: "fc567f22-d0d7-421f-8a05-5941f128ac44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:21.564350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.562391    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m\" (UniqueName: \"kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m\") pod \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") "
Nov 03 00:47:21.564350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.562483    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca\") pod \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\" (UID: \"09887ad0-66c2-48b5-9b75-34b6a4f4c313\") "
Nov 03 00:47:21.564350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.563759    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:21.564764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.564736    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca" pod "09887ad0-66c2-48b5-9b75-34b6a4f4c313" (UID: "09887ad0-66c2-48b5-9b75-34b6a4f4c313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:21.565140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.564838    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca" pod "09887ad0-66c2-48b5-9b75-34b6a4f4c313" (UID: "09887ad0-66c2-48b5-9b75-34b6a4f4c313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:21.571940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.571870    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca" pod "fc567f22-d0d7-421f-8a05-5941f128ac44" (UID: "fc567f22-d0d7-421f-8a05-5941f128ac44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:21.583844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.583808    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m" pod "09887ad0-66c2-48b5-9b75-34b6a4f4c313" (UID: "09887ad0-66c2-48b5-9b75-34b6a4f4c313"). InnerVolumeSpecName "kube-api-access-npf8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:21.586021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.585993    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74" pod "fc567f22-d0d7-421f-8a05-5941f128ac44" (UID: "fc567f22-d0d7-421f-8a05-5941f128ac44"). InnerVolumeSpecName "kube-api-access-hjl74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:21.665180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.665139    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:21.665180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.665181    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npf8m\" (UniqueName: \"kubernetes.io/projected/09887ad0-66c2-48b5-9b75-34b6a4f4c313-kube-api-access-npf8m\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:21.665422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.665198    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09887ad0-66c2-48b5-9b75-34b6a4f4c313-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:21.665422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.665211    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjl74\" (UniqueName: \"kubernetes.io/projected/fc567f22-d0d7-421f-8a05-5941f128ac44-kube-api-access-hjl74\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:21.665422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:21.665224    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc567f22-d0d7-421f-8a05-5941f128ac44-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:22.218819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.218717    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"
Nov 03 00:47:22.218819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.218742    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p" event={"ID":"fc567f22-d0d7-421f-8a05-5941f128ac44","Type":"ContainerDied","Data":"0a7769ba280ce3fa41152688c31d8b9f5ca6ae2377592d2ca5abfbe466547806"}
Nov 03 00:47:22.218819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.218793    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a7769ba280ce3fa41152688c31d8b9f5ca6ae2377592d2ca5abfbe466547806"
Nov 03 00:47:22.220341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.220316    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6" event={"ID":"09887ad0-66c2-48b5-9b75-34b6a4f4c313","Type":"ContainerDied","Data":"5fa833bcd369378a1f7637fd038ebdc7e6bf878e8de6c9dfae4c02e3672121f8"}
Nov 03 00:47:22.220460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.220350    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"
Nov 03 00:47:22.220460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.220361    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa833bcd369378a1f7637fd038ebdc7e6bf878e8de6c9dfae4c02e3672121f8"
Nov 03 00:47:22.572567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:22.572462    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:24.280263 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:24.280231    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:24.725739 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:47:24.725706    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:47:25.950554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:25.950515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:26.370618 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:26.370522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:26.503466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:26.502920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:26.816175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:26.816139    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:27.644230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:27.644191    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:28.040641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:28.040593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:29.361056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:29.361024    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:29.795360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:29.795316    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:29.893357 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:29.893325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:30.517527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.517492    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"]
Nov 03 00:47:30.522450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.519050    4550 cpu_manager.go:414] "containerMap: removing container" podUID="09887ad0-66c2-48b5-9b75-34b6a4f4c313" containerName="revoke-keys"
Nov 03 00:47:30.522691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.522669    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09887ad0-66c2-48b5-9b75-34b6a4f4c313" containerName="revoke-keys"
Nov 03 00:47:30.522791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.522782    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fc567f22-d0d7-421f-8a05-5941f128ac44" containerName="revoke-keys"
Nov 03 00:47:30.522860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.522851    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc567f22-d0d7-421f-8a05-5941f128ac44" containerName="revoke-keys"
Nov 03 00:47:30.523144 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.523129    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="09887ad0-66c2-48b5-9b75-34b6a4f4c313" containerName="revoke-keys"
Nov 03 00:47:30.523258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.523248    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc567f22-d0d7-421f-8a05-5941f128ac44" containerName="revoke-keys"
Nov 03 00:47:30.540285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.539634    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.544659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.544629    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"]
Nov 03 00:47:30.591504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.589716    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:47:30.603949 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.603619    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:47:30.609993 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.609509    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"]
Nov 03 00:47:30.617334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.615896    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"]
Nov 03 00:47:30.617334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.616837    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.624634 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.624606    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njlcv\" (UniqueName: \"kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.624789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.624710    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.624869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.624791    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.726087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726050    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.726273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726102    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njlcv\" (UniqueName: \"kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.726273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.726364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726324    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.726409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726361    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.726491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726465    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv5s\" (UniqueName: \"kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.726990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.726967    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.728692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.728648    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.745835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.745811    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njlcv\" (UniqueName: \"kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv\") pod \"maas-api-revoke-sub-f454281d-dsqtp\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.828826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.828719    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.829015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.828824    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv5s\" (UniqueName: \"kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.829075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.829045    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.830010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.829982    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.833008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.832985    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.853233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.853204    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv5s\" (UniqueName: \"kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s\") pod \"maas-api-revoke-sub-9e69121a-jpw8s\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:30.888214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.888177    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:30.935048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:30.935012    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:31.199843 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:47:31.199814    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1969f873_4481_4720_b8a8_f205af215e88.slice/crio-3769e1bd65691932a9f2209681b5df25150dbfff370d60b39db9cab29c67a5e4 WatchSource:0}: Error finding container 3769e1bd65691932a9f2209681b5df25150dbfff370d60b39db9cab29c67a5e4: Status 404 returned error can't find the container with id 3769e1bd65691932a9f2209681b5df25150dbfff370d60b39db9cab29c67a5e4
Nov 03 00:47:31.203092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:31.203063    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"]
Nov 03 00:47:31.256387 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:31.255826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:31.256387 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:47:31.256330    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b723a2_13ef_43ce_a3bf_b1e502091dd8.slice/crio-07334c7df477e435455ccb447b5e94ef17bc6a0dc102b5778bc1d3b293e66da7 WatchSource:0}: Error finding container 07334c7df477e435455ccb447b5e94ef17bc6a0dc102b5778bc1d3b293e66da7: Status 404 returned error can't find the container with id 07334c7df477e435455ccb447b5e94ef17bc6a0dc102b5778bc1d3b293e66da7
Nov 03 00:47:31.258762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:31.258737    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"]
Nov 03 00:47:31.282240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:31.282202    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s" event={"ID":"30b723a2-13ef-43ce-a3bf-b1e502091dd8","Type":"ContainerStarted","Data":"07334c7df477e435455ccb447b5e94ef17bc6a0dc102b5778bc1d3b293e66da7"}
Nov 03 00:47:31.283621 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:31.283599    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp" event={"ID":"1969f873-4481-4720-b8a8-f205af215e88","Type":"ContainerStarted","Data":"3769e1bd65691932a9f2209681b5df25150dbfff370d60b39db9cab29c67a5e4"}
Nov 03 00:47:32.295026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.294987    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s" event={"ID":"30b723a2-13ef-43ce-a3bf-b1e502091dd8","Type":"ContainerStarted","Data":"4e730e2872c89e60374430949907145ead3be9e6726fb532a01de7c380bc4d66"}
Nov 03 00:47:32.297331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.297304    4550 generic.go:356] "Generic (PLEG): container finished" podID="1969f873-4481-4720-b8a8-f205af215e88" containerID="0b2f40a533782c28d5e595337aae2a74d21669f06e1fcae7d8ad4bd0db4e9534" exitCode=0
Nov 03 00:47:32.297479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.297386    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp" event={"ID":"1969f873-4481-4720-b8a8-f205af215e88","Type":"ContainerDied","Data":"0b2f40a533782c28d5e595337aae2a74d21669f06e1fcae7d8ad4bd0db4e9534"}
Nov 03 00:47:32.336116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.336061    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s" podStartSLOduration=2.336048198 podStartE2EDuration="2.336048198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:32.334209937 +0000 UTC m=+2534.655746759" watchObservedRunningTime="2026-11-03 00:47:32.336048198 +0000 UTC m=+2534.657584996"
Nov 03 00:47:32.579556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.579416    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:32.671954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.671916    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:32.767060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.767022    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"]
Nov 03 00:47:32.771332 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.771307    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.791798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.791759    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"]
Nov 03 00:47:32.863310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.863216    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.863310 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.863263    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lhg\" (UniqueName: \"kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.863528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.863367    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.928594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.928558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:32.965560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.965137    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.965560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.965197    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lhg\" (UniqueName: \"kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.965560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.965262    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.966183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.966142    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:32.968388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:32.968357    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:33.016244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.016166    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lhg\" (UniqueName: \"kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg\") pod \"maas-api-revoke-sub-5e6dc5ee-9cwbw\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:33.091691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.091652    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:33.307582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.307545    4550 generic.go:356] "Generic (PLEG): container finished" podID="30b723a2-13ef-43ce-a3bf-b1e502091dd8" containerID="4e730e2872c89e60374430949907145ead3be9e6726fb532a01de7c380bc4d66" exitCode=0
Nov 03 00:47:33.308036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.307630    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s" event={"ID":"30b723a2-13ef-43ce-a3bf-b1e502091dd8","Type":"ContainerDied","Data":"4e730e2872c89e60374430949907145ead3be9e6726fb532a01de7c380bc4d66"}
Nov 03 00:47:33.420424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.420385    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"]
Nov 03 00:47:33.483746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.483717    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:33.578040 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.577999    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv\" (UniqueName: \"kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv\") pod \"1969f873-4481-4720-b8a8-f205af215e88\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") "
Nov 03 00:47:33.578225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.578202    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca\") pod \"1969f873-4481-4720-b8a8-f205af215e88\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") "
Nov 03 00:47:33.578842 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.578820    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca\") pod \"1969f873-4481-4720-b8a8-f205af215e88\" (UID: \"1969f873-4481-4720-b8a8-f205af215e88\") "
Nov 03 00:47:33.579630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.579604    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca" pod "1969f873-4481-4720-b8a8-f205af215e88" (UID: "1969f873-4481-4720-b8a8-f205af215e88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:33.580915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.580894    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca" pod "1969f873-4481-4720-b8a8-f205af215e88" (UID: "1969f873-4481-4720-b8a8-f205af215e88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:33.581240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.581221    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv" pod "1969f873-4481-4720-b8a8-f205af215e88" (UID: "1969f873-4481-4720-b8a8-f205af215e88"). InnerVolumeSpecName "kube-api-access-njlcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:33.681363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.681278    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1969f873-4481-4720-b8a8-f205af215e88-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:33.681363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.681315    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1969f873-4481-4720-b8a8-f205af215e88-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:33.681363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:33.681331    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njlcv\" (UniqueName: \"kubernetes.io/projected/1969f873-4481-4720-b8a8-f205af215e88-kube-api-access-njlcv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:34.316277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.316168    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw" event={"ID":"d9f4bf15-547c-495f-95e8-1c64b6f1dadb","Type":"ContainerStarted","Data":"da37f800fb23b980c2e87a5d6ebe48f45834a929e5d4a59e0331cf76663a7d0f"}
Nov 03 00:47:34.316277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.316232    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw" event={"ID":"d9f4bf15-547c-495f-95e8-1c64b6f1dadb","Type":"ContainerStarted","Data":"13ed0d8aa03223b4a31003128c233b09a24f3b761b4e8196c1b494290faf3c65"}
Nov 03 00:47:34.317569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.317548    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"
Nov 03 00:47:34.317693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.317585    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp" event={"ID":"1969f873-4481-4720-b8a8-f205af215e88","Type":"ContainerDied","Data":"3769e1bd65691932a9f2209681b5df25150dbfff370d60b39db9cab29c67a5e4"}
Nov 03 00:47:34.317693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.317633    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3769e1bd65691932a9f2209681b5df25150dbfff370d60b39db9cab29c67a5e4"
Nov 03 00:47:34.361614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.361550    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw" podStartSLOduration=2.361529337 podStartE2EDuration="2.361529337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:34.356620708 +0000 UTC m=+2536.678157503" watchObservedRunningTime="2026-11-03 00:47:34.361529337 +0000 UTC m=+2536.683066137"
Nov 03 00:47:34.565988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.565962    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:34.695924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.695889    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:34.697065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.697043    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca\") pod \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") "
Nov 03 00:47:34.697163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.697104    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca\") pod \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") "
Nov 03 00:47:34.697302 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.697286    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s\" (UniqueName: \"kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s\") pod \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\" (UID: \"30b723a2-13ef-43ce-a3bf-b1e502091dd8\") "
Nov 03 00:47:34.697558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.697528    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca" pod "30b723a2-13ef-43ce-a3bf-b1e502091dd8" (UID: "30b723a2-13ef-43ce-a3bf-b1e502091dd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:34.697889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.697865    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:34.699351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.699331    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca" pod "30b723a2-13ef-43ce-a3bf-b1e502091dd8" (UID: "30b723a2-13ef-43ce-a3bf-b1e502091dd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:34.699533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.699346    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s" pod "30b723a2-13ef-43ce-a3bf-b1e502091dd8" (UID: "30b723a2-13ef-43ce-a3bf-b1e502091dd8"). InnerVolumeSpecName "kube-api-access-bnv5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:34.799356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.799319    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv5s\" (UniqueName: \"kubernetes.io/projected/30b723a2-13ef-43ce-a3bf-b1e502091dd8-kube-api-access-bnv5s\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:34.799356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:34.799352    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b723a2-13ef-43ce-a3bf-b1e502091dd8-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:35.326318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:35.326283    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"
Nov 03 00:47:35.326818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:35.326315    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s" event={"ID":"30b723a2-13ef-43ce-a3bf-b1e502091dd8","Type":"ContainerDied","Data":"07334c7df477e435455ccb447b5e94ef17bc6a0dc102b5778bc1d3b293e66da7"}
Nov 03 00:47:35.326818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:35.326376    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07334c7df477e435455ccb447b5e94ef17bc6a0dc102b5778bc1d3b293e66da7"
Nov 03 00:47:35.328022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:35.327996    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9f4bf15-547c-495f-95e8-1c64b6f1dadb" containerID="da37f800fb23b980c2e87a5d6ebe48f45834a929e5d4a59e0331cf76663a7d0f" exitCode=0
Nov 03 00:47:35.328173 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:35.328056    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw" event={"ID":"d9f4bf15-547c-495f-95e8-1c64b6f1dadb","Type":"ContainerDied","Data":"da37f800fb23b980c2e87a5d6ebe48f45834a929e5d4a59e0331cf76663a7d0f"}
Nov 03 00:47:36.516614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.516589    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:36.543965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.543933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:36.647688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.647590    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca\") pod \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") "
Nov 03 00:47:36.647688 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.647651    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg\" (UniqueName: \"kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg\") pod \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") "
Nov 03 00:47:36.647941 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.647789    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca\") pod \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\" (UID: \"d9f4bf15-547c-495f-95e8-1c64b6f1dadb\") "
Nov 03 00:47:36.648211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.648179    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca" pod "d9f4bf15-547c-495f-95e8-1c64b6f1dadb" (UID: "d9f4bf15-547c-495f-95e8-1c64b6f1dadb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:36.648523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.648506    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:36.651225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.651201    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg" pod "d9f4bf15-547c-495f-95e8-1c64b6f1dadb" (UID: "d9f4bf15-547c-495f-95e8-1c64b6f1dadb"). InnerVolumeSpecName "kube-api-access-84lhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:36.651321 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.651246    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca" pod "d9f4bf15-547c-495f-95e8-1c64b6f1dadb" (UID: "d9f4bf15-547c-495f-95e8-1c64b6f1dadb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:36.749992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.749956    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:36.749992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:36.749987    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84lhg\" (UniqueName: \"kubernetes.io/projected/d9f4bf15-547c-495f-95e8-1c64b6f1dadb-kube-api-access-84lhg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:37.306213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:37.306166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:37.345068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:37.345038    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"
Nov 03 00:47:37.345068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:37.345056    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw" event={"ID":"d9f4bf15-547c-495f-95e8-1c64b6f1dadb","Type":"ContainerDied","Data":"13ed0d8aa03223b4a31003128c233b09a24f3b761b4e8196c1b494290faf3c65"}
Nov 03 00:47:37.345295 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:37.345101    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13ed0d8aa03223b4a31003128c233b09a24f3b761b4e8196c1b494290faf3c65"
Nov 03 00:47:37.406788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:37.406750    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:38.198617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:38.198580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:39.973250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:39.973210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:41.072103 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:41.072055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:41.172468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:41.172411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:41.628628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:41.628584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:41.752475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:41.749650    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:41.840270 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:41.840232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:42.039470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:42.038354    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:42.132834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:42.132788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:42.386506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:42.382238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:42.675319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:42.675281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:42.775014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:42.774978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:43.517248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:43.517212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:44.362421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.362386    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"]
Nov 03 00:47:44.363019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363004    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9f4bf15-547c-495f-95e8-1c64b6f1dadb" containerName="revoke-keys"
Nov 03 00:47:44.363064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363021    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f4bf15-547c-495f-95e8-1c64b6f1dadb" containerName="revoke-keys"
Nov 03 00:47:44.363064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363053    4550 cpu_manager.go:414] "containerMap: removing container" podUID="30b723a2-13ef-43ce-a3bf-b1e502091dd8" containerName="revoke-keys"
Nov 03 00:47:44.363064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363058    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b723a2-13ef-43ce-a3bf-b1e502091dd8" containerName="revoke-keys"
Nov 03 00:47:44.363211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363068    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1969f873-4481-4720-b8a8-f205af215e88" containerName="revoke-keys"
Nov 03 00:47:44.363211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363073    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1969f873-4481-4720-b8a8-f205af215e88" containerName="revoke-keys"
Nov 03 00:47:44.363211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363172    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f4bf15-547c-495f-95e8-1c64b6f1dadb" containerName="revoke-keys"
Nov 03 00:47:44.363211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363184    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b723a2-13ef-43ce-a3bf-b1e502091dd8" containerName="revoke-keys"
Nov 03 00:47:44.363211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.363191    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1969f873-4481-4720-b8a8-f205af215e88" containerName="revoke-keys"
Nov 03 00:47:44.366953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.366933    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.384863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.384836    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"]
Nov 03 00:47:44.393377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.393328    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:47:44.393677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.393349    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:47:44.398570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.398542    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.398570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.398714    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.398570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.398804    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zj7b\" (UniqueName: \"kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.500488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.500442    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.500708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.500610    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.500708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.500651    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zj7b\" (UniqueName: \"kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.501279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.501258    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.502819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.502801    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.520966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.520940    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zj7b\" (UniqueName: \"kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b\") pod \"maas-api-revoke-sub-6e9793a6-t74nm\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.683463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.683404    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:44.989331 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:47:44.989298    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb699652_7ada_46e0_9baa_a3f9fe9c957e.slice/crio-f1c2ee0878c70c58212ee5c5597b5e4b3c45c6dbd8f4aceabfd42e9a8a6be600 WatchSource:0}: Error finding container f1c2ee0878c70c58212ee5c5597b5e4b3c45c6dbd8f4aceabfd42e9a8a6be600: Status 404 returned error can't find the container with id f1c2ee0878c70c58212ee5c5597b5e4b3c45c6dbd8f4aceabfd42e9a8a6be600
Nov 03 00:47:44.993181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:44.993151    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"]
Nov 03 00:47:45.151367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.151328    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"]
Nov 03 00:47:45.156862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.156837    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.177138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.177110    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"]
Nov 03 00:47:45.211137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.211104    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.211402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.211382    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.211548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.211531    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjg5t\" (UniqueName: \"kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.219676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.219645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:45.314451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.313747    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.314451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.313854    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjg5t\" (UniqueName: \"kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.314451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.313970    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.316714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.315295    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.318391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.318362    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.334066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.334040    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjg5t\" (UniqueName: \"kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t\") pod \"maas-api-revoke-sub-e1d8715f-t622f\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.444693 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.444654    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm" event={"ID":"eb699652-7ada-46e0-9baa-a3f9fe9c957e","Type":"ContainerStarted","Data":"6895376fcb2d024cedcbdb527a5b3d1f7879fc4ad3f10a7a51103944d83fc791"}
Nov 03 00:47:45.444903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.444732    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm" event={"ID":"eb699652-7ada-46e0-9baa-a3f9fe9c957e","Type":"ContainerStarted","Data":"f1c2ee0878c70c58212ee5c5597b5e4b3c45c6dbd8f4aceabfd42e9a8a6be600"}
Nov 03 00:47:45.482912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.482883    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:45.486119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.485948    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm" podStartSLOduration=1.485935462 podStartE2EDuration="1.485935462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:45.482884699 +0000 UTC m=+2547.804421497" watchObservedRunningTime="2026-11-03 00:47:45.485935462 +0000 UTC m=+2547.807472260"
Nov 03 00:47:45.789017 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:47:45.788970    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8a47ab_6458_44bd_a402_782199c96f10.slice/crio-f912eabad97e8a4b33993857090ce723df47255c64a7538159541b6e93526bac WatchSource:0}: Error finding container f912eabad97e8a4b33993857090ce723df47255c64a7538159541b6e93526bac: Status 404 returned error can't find the container with id f912eabad97e8a4b33993857090ce723df47255c64a7538159541b6e93526bac
Nov 03 00:47:45.790961 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:45.790919    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"]
Nov 03 00:47:46.452628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:46.452590    4550 generic.go:356] "Generic (PLEG): container finished" podID="eb699652-7ada-46e0-9baa-a3f9fe9c957e" containerID="6895376fcb2d024cedcbdb527a5b3d1f7879fc4ad3f10a7a51103944d83fc791" exitCode=0
Nov 03 00:47:46.452831 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:46.452674    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm" event={"ID":"eb699652-7ada-46e0-9baa-a3f9fe9c957e","Type":"ContainerDied","Data":"6895376fcb2d024cedcbdb527a5b3d1f7879fc4ad3f10a7a51103944d83fc791"}
Nov 03 00:47:46.454251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:46.454225    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f" event={"ID":"8b8a47ab-6458-44bd-a402-782199c96f10","Type":"ContainerStarted","Data":"fdf8141e2a057bbc5822b2201fa44d98d43e831f8b0d5e2e47a995f165d3bb11"}
Nov 03 00:47:46.454390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:46.454265    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f" event={"ID":"8b8a47ab-6458-44bd-a402-782199c96f10","Type":"ContainerStarted","Data":"f912eabad97e8a4b33993857090ce723df47255c64a7538159541b6e93526bac"}
Nov 03 00:47:46.537461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:46.533224    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f" podStartSLOduration=1.5332058929999999 podStartE2EDuration="1.533205893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:46.531738633 +0000 UTC m=+2548.853275432" watchObservedRunningTime="2026-11-03 00:47:46.533205893 +0000 UTC m=+2548.854742693"
Nov 03 00:47:46.841615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:46.841579    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:47.464564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.464526    4550 generic.go:356] "Generic (PLEG): container finished" podID="8b8a47ab-6458-44bd-a402-782199c96f10" containerID="fdf8141e2a057bbc5822b2201fa44d98d43e831f8b0d5e2e47a995f165d3bb11" exitCode=0
Nov 03 00:47:47.464747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.464608    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f" event={"ID":"8b8a47ab-6458-44bd-a402-782199c96f10","Type":"ContainerDied","Data":"fdf8141e2a057bbc5822b2201fa44d98d43e831f8b0d5e2e47a995f165d3bb11"}
Nov 03 00:47:47.656902 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.656872    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:47.748679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.748586    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b\" (UniqueName: \"kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b\") pod \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") "
Nov 03 00:47:47.748864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.748703    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca\") pod \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") "
Nov 03 00:47:47.748864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.748761    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca\") pod \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\" (UID: \"eb699652-7ada-46e0-9baa-a3f9fe9c957e\") "
Nov 03 00:47:47.749081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.749056    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca" pod "eb699652-7ada-46e0-9baa-a3f9fe9c957e" (UID: "eb699652-7ada-46e0-9baa-a3f9fe9c957e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:47.749360 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.749342    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.750781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.750760    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b" pod "eb699652-7ada-46e0-9baa-a3f9fe9c957e" (UID: "eb699652-7ada-46e0-9baa-a3f9fe9c957e"). InnerVolumeSpecName "kube-api-access-8zj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:47.750851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.750815    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca" pod "eb699652-7ada-46e0-9baa-a3f9fe9c957e" (UID: "eb699652-7ada-46e0-9baa-a3f9fe9c957e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:47.851251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.851207    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zj7b\" (UniqueName: \"kubernetes.io/projected/eb699652-7ada-46e0-9baa-a3f9fe9c957e-kube-api-access-8zj7b\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.851251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:47.851240    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb699652-7ada-46e0-9baa-a3f9fe9c957e-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:48.347593 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.347558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:48.444018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.443978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:48.473692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.473660    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"
Nov 03 00:47:48.473879 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.473671    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm" event={"ID":"eb699652-7ada-46e0-9baa-a3f9fe9c957e","Type":"ContainerDied","Data":"f1c2ee0878c70c58212ee5c5597b5e4b3c45c6dbd8f4aceabfd42e9a8a6be600"}
Nov 03 00:47:48.475406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.475381    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1c2ee0878c70c58212ee5c5597b5e4b3c45c6dbd8f4aceabfd42e9a8a6be600"
Nov 03 00:47:48.559034 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.558972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:48.729959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.729920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:48.760680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.760650    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:48.925027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.924991    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca\") pod \"8b8a47ab-6458-44bd-a402-782199c96f10\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") "
Nov 03 00:47:48.925410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.925041    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t\" (UniqueName: \"kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t\") pod \"8b8a47ab-6458-44bd-a402-782199c96f10\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") "
Nov 03 00:47:48.925410 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.925127    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca\") pod \"8b8a47ab-6458-44bd-a402-782199c96f10\" (UID: \"8b8a47ab-6458-44bd-a402-782199c96f10\") "
Nov 03 00:47:48.925530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.925402    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca" pod "8b8a47ab-6458-44bd-a402-782199c96f10" (UID: "8b8a47ab-6458-44bd-a402-782199c96f10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:48.925806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.925779    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:48.927334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.927311    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t" pod "8b8a47ab-6458-44bd-a402-782199c96f10" (UID: "8b8a47ab-6458-44bd-a402-782199c96f10"). InnerVolumeSpecName "kube-api-access-gjg5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:48.927334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:48.927320    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca" pod "8b8a47ab-6458-44bd-a402-782199c96f10" (UID: "8b8a47ab-6458-44bd-a402-782199c96f10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:47:49.027218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.027132    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjg5t\" (UniqueName: \"kubernetes.io/projected/8b8a47ab-6458-44bd-a402-782199c96f10-kube-api-access-gjg5t\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:49.027218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.027164    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b8a47ab-6458-44bd-a402-782199c96f10-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:47:49.074722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.074687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:49.179787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.179754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:49.280079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.279979    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:49.485277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.485244    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f" event={"ID":"8b8a47ab-6458-44bd-a402-782199c96f10","Type":"ContainerDied","Data":"f912eabad97e8a4b33993857090ce723df47255c64a7538159541b6e93526bac"}
Nov 03 00:47:49.485481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.485293    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f912eabad97e8a4b33993857090ce723df47255c64a7538159541b6e93526bac"
Nov 03 00:47:49.485481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:49.485296    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"
Nov 03 00:47:50.300641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:50.300606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:52.040373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:52.040335    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:53.641703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:53.641669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:54.730953 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:47:54.730917    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:47:54.984776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:54.984657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:55.135577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:55.135540    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:55.235268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:55.235183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:55.470835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:55.470791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:55.637708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:55.637617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:55.815780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:55.795009    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:55.930454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:55.929623    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:56.297860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:56.297782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:56.591286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:56.591198    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:56.694712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:56.694677    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:56.802575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:56.802538    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:56.934892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:56.934859    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.225520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.223918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.365465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.362973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.494515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.494400    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.606483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.606450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.712824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.712788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.840542 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.840453    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:57.963485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:57.963455    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.214453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.208053    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.334611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.334577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.437689 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.437656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.538641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.538543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.644996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.644924    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.748339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.748241    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:58.950675 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:58.950546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:59.481970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:59.481823    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:59.693676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:59.693627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:47:59.839388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:47:59.837500    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:00.371763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:00.371731    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:01.202106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:01.202073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:01.302727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:01.302672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:01.524208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:01.524106    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:01.630052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:01.630011    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:02.174984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:02.174952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:02.502738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:02.502636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:02.607605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:02.607571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:02.823513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:02.816100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:02.950864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:02.950826    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:03.054830 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:03.054798    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:03.592886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:03.592835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:03.808525 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:03.808471    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:04.547724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:04.547682    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:05.437770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:05.437736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:06.187348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:06.187310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:07.116580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:07.116543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:08.337695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:08.337657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:09.040900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:09.040860    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:09.626600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:09.626569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:09.746381 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:09.746336    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:10.055761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:10.055725    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:10.276573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:10.276525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:10.888992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:10.888956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:11.008004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:11.007962    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:12.339258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:12.339224    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:12.694878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:12.694835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:14.310135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:14.310097    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:16.027314 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:16.027279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:17.624945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:17.624911    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:19.302658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:19.302618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:19.726717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:19.726687    4550 scope.go:122] "RemoveContainer" containerID="6567494d5c238f65d6d0693f019aea036cfdf764a39ea23d58ff89edaec941ab"
Nov 03 00:48:19.736392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:19.736370    4550 scope.go:122] "RemoveContainer" containerID="a1c61fcd37e1faa5339d5b69444a037ca6bdef3f82e0666bb5f6d845ad5f6c08"
Nov 03 00:48:19.745369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:19.745350    4550 scope.go:122] "RemoveContainer" containerID="2a350f91660ddb95175a8acb50911d088a2573d63f6198c73ea4a0f9f1ff8041"
Nov 03 00:48:19.754308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:19.754281    4550 scope.go:122] "RemoveContainer" containerID="2c3efa1f3e9114582e9523c81bf872678a688c6474b679b52bc50c9d4ef225d6"
Nov 03 00:48:20.909747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:20.909715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:22.637692 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:22.637649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:24.469904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:24.469867    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:24.724265 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:48:24.724172    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:48:26.227001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:26.226969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:28.211079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:28.211043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:28.866218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:28.866180    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:29.160028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.153277    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"]
Nov 03 00:48:29.161169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.161137    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8b8a47ab-6458-44bd-a402-782199c96f10" containerName="revoke-keys"
Nov 03 00:48:29.166690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.166658    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8a47ab-6458-44bd-a402-782199c96f10" containerName="revoke-keys"
Nov 03 00:48:29.166860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.166710    4550 cpu_manager.go:414] "containerMap: removing container" podUID="eb699652-7ada-46e0-9baa-a3f9fe9c957e" containerName="revoke-keys"
Nov 03 00:48:29.166860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.166718    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb699652-7ada-46e0-9baa-a3f9fe9c957e" containerName="revoke-keys"
Nov 03 00:48:29.167236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.167212    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8a47ab-6458-44bd-a402-782199c96f10" containerName="revoke-keys"
Nov 03 00:48:29.167328 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.167241    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb699652-7ada-46e0-9baa-a3f9fe9c957e" containerName="revoke-keys"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.242619    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245482    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245536    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245619    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245762    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwb6v\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245843    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245890    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.245943    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.246016    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.264455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.246114    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.585264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.585204    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.586477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.585948    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.586477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.586029    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.586477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.586095    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.586896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.586871    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwb6v\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.606091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.587012    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.606091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.592300    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"]
Nov 03 00:48:29.606483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.605631    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.607264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.607243    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.607793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.607764    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.607992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.607978    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.608211 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.608195    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.609112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.609089    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.609254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.607136    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.609624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.609594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.689645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.688594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.689645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.689490    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwb6v\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.694084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.693920    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.694084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.694031    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:29.906836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:29.906802    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:30.219983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:30.219883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:30.227701 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:48:30.227672    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cb3db_89da_44c7_b194_5150ec82a1d8.slice/crio-752f0a1b33cc06a2acec09eda925d23dd2f973db3efaa2ef7c5012fd80e14750 WatchSource:0}: Error finding container 752f0a1b33cc06a2acec09eda925d23dd2f973db3efaa2ef7c5012fd80e14750: Status 404 returned error can't find the container with id 752f0a1b33cc06a2acec09eda925d23dd2f973db3efaa2ef7c5012fd80e14750
Nov 03 00:48:30.229110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:30.229084    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"]
Nov 03 00:48:30.230649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:30.230618    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:30.230735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:30.230684    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:30.230735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:30.230711    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:48:31.111025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:31.110985    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" event={"ID":"2b6cb3db-89da-44c7-b194-5150ec82a1d8","Type":"ContainerStarted","Data":"9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b"}
Nov 03 00:48:31.111586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:31.111059    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" event={"ID":"2b6cb3db-89da-44c7-b194-5150ec82a1d8","Type":"ContainerStarted","Data":"752f0a1b33cc06a2acec09eda925d23dd2f973db3efaa2ef7c5012fd80e14750"}
Nov 03 00:48:31.183453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:31.179661    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podStartSLOduration=2.179638798 podStartE2EDuration="2.179638798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:31.174936727 +0000 UTC m=+2593.496473528" watchObservedRunningTime="2026-11-03 00:48:31.179638798 +0000 UTC m=+2593.501175600"
Nov 03 00:48:31.907937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:31.907906    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:31.909399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:31.909367    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:48:31.909562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:31.909459    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 00:48:32.111602 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:32.111563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:32.907349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:32.907299    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:48:32.907560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:32.907371    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 00:48:33.854796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:33.854757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:33.907426 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:33.907388    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:48:33.907672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:33.907468    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 00:48:34.907349 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:34.907312    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:48:34.907778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:34.907366    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 00:48:35.659457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:35.656741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:35.907304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:35.907265    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:48:35.907502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:35.907333    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 00:48:36.912998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:36.912967    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:36.913496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:36.913456    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:36.914305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:36.914278    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 00:48:37.322804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:37.322704    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:39.162104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:39.162070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:40.832227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:40.832187    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:42.506610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:42.506571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:44.178565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:44.178529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:45.930405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:45.930370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:47.777900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:47.777868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:49.290768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:49.290727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:50.014411 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:50.014376    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:50.113553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:50.113510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:50.949036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:50.949000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:51.048735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.048678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:51.148448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.148406    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:51.565324 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.565285    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"]
Nov 03 00:48:51.570000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.569972    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.593330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.593297    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"]
Nov 03 00:48:51.601681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.601650    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:48:51.601857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.601721    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:48:51.611794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.611764    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.611986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.611865    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.612203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.612184    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5cl\" (UniqueName: \"kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.713856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.713819    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5cl\" (UniqueName: \"kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.714068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.713929    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.714068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.714017    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.714717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.714690    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.716424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.716399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.735009 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.734980    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5cl\" (UniqueName: \"kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl\") pod \"maas-api-revoke-sub-8e0da59f-lb9tp\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:51.891981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:51.891886    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:52.198155 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:48:52.198115    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c1c337_d01f_4cec_a974_3a74ab12bce5.slice/crio-5d6318d1751bb63ca0e8d51ca4ec47996c4ac4dbd5457a96630ca18a07b276c9 WatchSource:0}: Error finding container 5d6318d1751bb63ca0e8d51ca4ec47996c4ac4dbd5457a96630ca18a07b276c9: Status 404 returned error can't find the container with id 5d6318d1751bb63ca0e8d51ca4ec47996c4ac4dbd5457a96630ca18a07b276c9
Nov 03 00:48:52.201742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:52.201714    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"]
Nov 03 00:48:52.273509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:52.273473    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp" event={"ID":"41c1c337-d01f-4cec-a974-3a74ab12bce5","Type":"ContainerStarted","Data":"5d6318d1751bb63ca0e8d51ca4ec47996c4ac4dbd5457a96630ca18a07b276c9"}
Nov 03 00:48:52.924964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:52.924933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:53.284896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:53.284767    4550 generic.go:356] "Generic (PLEG): container finished" podID="41c1c337-d01f-4cec-a974-3a74ab12bce5" containerID="815cc974e4f54c1f2d409478f89309336e306d8800342627e17a5ef58c4950e6" exitCode=0
Nov 03 00:48:53.285375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:53.285007    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp" event={"ID":"41c1c337-d01f-4cec-a974-3a74ab12bce5","Type":"ContainerDied","Data":"815cc974e4f54c1f2d409478f89309336e306d8800342627e17a5ef58c4950e6"}
Nov 03 00:48:54.481481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.481450    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:54.560655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.559601    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl\" (UniqueName: \"kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl\") pod \"41c1c337-d01f-4cec-a974-3a74ab12bce5\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") "
Nov 03 00:48:54.560655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.559774    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca\") pod \"41c1c337-d01f-4cec-a974-3a74ab12bce5\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") "
Nov 03 00:48:54.560655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.559924    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca\") pod \"41c1c337-d01f-4cec-a974-3a74ab12bce5\" (UID: \"41c1c337-d01f-4cec-a974-3a74ab12bce5\") "
Nov 03 00:48:54.560976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.560866    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca" pod "41c1c337-d01f-4cec-a974-3a74ab12bce5" (UID: "41c1c337-d01f-4cec-a974-3a74ab12bce5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:54.563367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.563304    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca" pod "41c1c337-d01f-4cec-a974-3a74ab12bce5" (UID: "41c1c337-d01f-4cec-a974-3a74ab12bce5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:48:54.563545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.563508    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl" pod "41c1c337-d01f-4cec-a974-3a74ab12bce5" (UID: "41c1c337-d01f-4cec-a974-3a74ab12bce5"). InnerVolumeSpecName "kube-api-access-9n5cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:54.639396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.639352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:54.661826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.661796    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:48:54.661826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.661823    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n5cl\" (UniqueName: \"kubernetes.io/projected/41c1c337-d01f-4cec-a974-3a74ab12bce5-kube-api-access-9n5cl\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:48:54.661826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:54.661832    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41c1c337-d01f-4cec-a974-3a74ab12bce5-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:48:54.751186 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:48:54.751143    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:48:55.299493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:55.299458    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"
Nov 03 00:48:55.299493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:55.299473    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp" event={"ID":"41c1c337-d01f-4cec-a974-3a74ab12bce5","Type":"ContainerDied","Data":"5d6318d1751bb63ca0e8d51ca4ec47996c4ac4dbd5457a96630ca18a07b276c9"}
Nov 03 00:48:55.299717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:55.299528    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d6318d1751bb63ca0e8d51ca4ec47996c4ac4dbd5457a96630ca18a07b276c9"
Nov 03 00:48:56.337140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:56.337105    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:57.985599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:57.985564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:48:59.670468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:48:59.670424    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:01.426509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:01.426475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:02.953771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:02.953737    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:03.061465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.061055    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:03.370345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.370157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:03.630499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.626485    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"]
Nov 03 00:49:03.630499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.627495    4550 cpu_manager.go:414] "containerMap: removing container" podUID="41c1c337-d01f-4cec-a974-3a74ab12bce5" containerName="revoke-keys"
Nov 03 00:49:03.630499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.627516    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41c1c337-d01f-4cec-a974-3a74ab12bce5" containerName="revoke-keys"
Nov 03 00:49:03.630499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.627675    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="41c1c337-d01f-4cec-a974-3a74ab12bce5" containerName="revoke-keys"
Nov 03 00:49:03.660536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.657819    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"]
Nov 03 00:49:03.660536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.658024    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.682309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.681769    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:49:03.713854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.712450    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:49:03.755330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.747317    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.755330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.747425    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.755330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.747473    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn6ll\" (UniqueName: \"kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.848325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.848289    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.848325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.848324    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn6ll\" (UniqueName: \"kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.848638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.848515    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.849218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.849195    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.850716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.850698    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:03.868132 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:03.868103    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn6ll\" (UniqueName: \"kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll\") pod \"maas-api-revoke-sub-d44d572a-4lccw\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:04.129313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:04.129277    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:04.432207 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:49:04.432182    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dd7b05_7d9e_4853_90b6_7415e3496e31.slice/crio-192bd1c2f6cd2b4f303345ac59be99ce423c0ae03a91ccce436bef6f2cfa3244 WatchSource:0}: Error finding container 192bd1c2f6cd2b4f303345ac59be99ce423c0ae03a91ccce436bef6f2cfa3244: Status 404 returned error can't find the container with id 192bd1c2f6cd2b4f303345ac59be99ce423c0ae03a91ccce436bef6f2cfa3244
Nov 03 00:49:04.434528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:04.434501    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"]
Nov 03 00:49:05.055543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:05.055426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:05.371283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:05.371191    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw" event={"ID":"65dd7b05-7d9e-4853-90b6-7415e3496e31","Type":"ContainerStarted","Data":"54c0d65fe85fd982177d625954e7c2be11651c97bccb19075f2833026e6df30b"}
Nov 03 00:49:05.371283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:05.371248    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw" event={"ID":"65dd7b05-7d9e-4853-90b6-7415e3496e31","Type":"ContainerStarted","Data":"192bd1c2f6cd2b4f303345ac59be99ce423c0ae03a91ccce436bef6f2cfa3244"}
Nov 03 00:49:05.415072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:05.415019    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw" podStartSLOduration=2.415005274 podStartE2EDuration="2.415005274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:05.411313199 +0000 UTC m=+2627.732850016" watchObservedRunningTime="2026-11-03 00:49:05.415005274 +0000 UTC m=+2627.736542073"
Nov 03 00:49:06.380912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:06.380874    4550 generic.go:356] "Generic (PLEG): container finished" podID="65dd7b05-7d9e-4853-90b6-7415e3496e31" containerID="54c0d65fe85fd982177d625954e7c2be11651c97bccb19075f2833026e6df30b" exitCode=0
Nov 03 00:49:06.381323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:06.380961    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw" event={"ID":"65dd7b05-7d9e-4853-90b6-7415e3496e31","Type":"ContainerDied","Data":"54c0d65fe85fd982177d625954e7c2be11651c97bccb19075f2833026e6df30b"}
Nov 03 00:49:06.683147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:06.683076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:08.131152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.131120    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:08.262533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.262501    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca\") pod \"65dd7b05-7d9e-4853-90b6-7415e3496e31\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") "
Nov 03 00:49:08.262757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.262616    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca\") pod \"65dd7b05-7d9e-4853-90b6-7415e3496e31\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") "
Nov 03 00:49:08.262757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.262695    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll\" (UniqueName: \"kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll\") pod \"65dd7b05-7d9e-4853-90b6-7415e3496e31\" (UID: \"65dd7b05-7d9e-4853-90b6-7415e3496e31\") "
Nov 03 00:49:08.263052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.263025    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca" pod "65dd7b05-7d9e-4853-90b6-7415e3496e31" (UID: "65dd7b05-7d9e-4853-90b6-7415e3496e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:08.263260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.263239    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:08.264684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.264655    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca" pod "65dd7b05-7d9e-4853-90b6-7415e3496e31" (UID: "65dd7b05-7d9e-4853-90b6-7415e3496e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:49:08.264773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.264742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll" pod "65dd7b05-7d9e-4853-90b6-7415e3496e31" (UID: "65dd7b05-7d9e-4853-90b6-7415e3496e31"). InnerVolumeSpecName "kube-api-access-dn6ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:08.365133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.365039    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dd7b05-7d9e-4853-90b6-7415e3496e31-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:08.365133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.365082    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn6ll\" (UniqueName: \"kubernetes.io/projected/65dd7b05-7d9e-4853-90b6-7415e3496e31-kube-api-access-dn6ll\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:08.420184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.420147    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"
Nov 03 00:49:08.420184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.420172    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw" event={"ID":"65dd7b05-7d9e-4853-90b6-7415e3496e31","Type":"ContainerDied","Data":"192bd1c2f6cd2b4f303345ac59be99ce423c0ae03a91ccce436bef6f2cfa3244"}
Nov 03 00:49:08.420472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.420218    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="192bd1c2f6cd2b4f303345ac59be99ce423c0ae03a91ccce436bef6f2cfa3244"
Nov 03 00:49:08.651406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:08.651371    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:10.228212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:10.228184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:11.247084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.247050    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"]
Nov 03 00:49:11.247706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.247690    4550 cpu_manager.go:414] "containerMap: removing container" podUID="65dd7b05-7d9e-4853-90b6-7415e3496e31" containerName="revoke-keys"
Nov 03 00:49:11.247756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.247707    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dd7b05-7d9e-4853-90b6-7415e3496e31" containerName="revoke-keys"
Nov 03 00:49:11.247818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.247809    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dd7b05-7d9e-4853-90b6-7415e3496e31" containerName="revoke-keys"
Nov 03 00:49:11.251529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.251505    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.279395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.279364    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:49:11.279613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.279462    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:49:11.293482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.293418    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"]
Nov 03 00:49:11.308499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.308461    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.308656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.308642    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.308728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.308689    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.410134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.410086    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.410350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.410275    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.410473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.410360    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.411906 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.411876    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.413745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.413717    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.437239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.437204    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726\") pod \"maas-api-revoke-sub-8b271f0c-vrwd6\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:11.441449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.441404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:11.537234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.537143    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:11.580582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:11.580550    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:12.115485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.113104    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"]
Nov 03 00:49:12.289079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.289045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:12.455472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.455416    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6" event={"ID":"77f51d6e-439d-4a08-be58-32a14c67cac2","Type":"ContainerStarted","Data":"b3a37a2f2188b34780edc3dbf720a497cd5ef15784e42a9eff5ba4aa2ef1e65d"}
Nov 03 00:49:12.472880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.472852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:12.774777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.774746    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"]
Nov 03 00:49:12.779026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.779007    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.795389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.795021    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-35649651d663\""
Nov 03 00:49:12.802479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.799814    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"]
Nov 03 00:49:12.807859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.807840    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-0219878e6cb3\""
Nov 03 00:49:12.807998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.807934    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"routing-overlay\""
Nov 03 00:49:12.808078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.808065    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"kube-root-ca.crt\""
Nov 03 00:49:12.808145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.808103    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-955dc3\""
Nov 03 00:49:12.808200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.808171    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-e2e--70281560-dockercfg-lrv8c\""
Nov 03 00:49:12.808255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.808239    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"openshift-service-ca.crt\""
Nov 03 00:49:12.855195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.855161    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.855371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.855345    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5h4\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.855690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.855664    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.855785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.855746    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.905331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.905200    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:12.958267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.957368    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.958267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.957513    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5h4\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.958267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.957663    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.958267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.957735    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.958810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.958780    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.959598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.959574    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.980480 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.977887    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5h4\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:12.981522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:12.981300    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:13.005483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.005383    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:13.094890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.094851    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:13.344509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.342112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:13.435091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.434903    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:13.492075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.492037    4550 generic.go:356] "Generic (PLEG): container finished" podID="77f51d6e-439d-4a08-be58-32a14c67cac2" containerID="0b0dadd40158588c693ea5bb2a8d8a1d267ba84dfa71a5f6ff9738b1d9568326" exitCode=0
Nov 03 00:49:13.492645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.492614    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6" event={"ID":"77f51d6e-439d-4a08-be58-32a14c67cac2","Type":"ContainerDied","Data":"0b0dadd40158588c693ea5bb2a8d8a1d267ba84dfa71a5f6ff9738b1d9568326"}
Nov 03 00:49:13.999472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:13.999425    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"]
Nov 03 00:49:14.489498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.489465    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"]
Nov 03 00:49:14.508081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.508041    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw" event={"ID":"4ab7bcac-76bf-4e00-a320-75f5831b69a3","Type":"ContainerStarted","Data":"47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505"}
Nov 03 00:49:14.508289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.508142    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw" event={"ID":"4ab7bcac-76bf-4e00-a320-75f5831b69a3","Type":"ContainerStarted","Data":"094bfa2e7b6d5939a7913512806d850df44d132e9be204802ba182546a3113b3"}
Nov 03 00:49:14.511998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.511970    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:14.581453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.574255    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw" podStartSLOduration=2.574232276 podStartE2EDuration="2.574232276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:14.565166372 +0000 UTC m=+2636.886703192" watchObservedRunningTime="2026-11-03 00:49:14.574232276 +0000 UTC m=+2636.895769075"
Nov 03 00:49:14.745389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.745315    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:14.769580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.769545    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca\") pod \"77f51d6e-439d-4a08-be58-32a14c67cac2\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") "
Nov 03 00:49:14.769799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.769703    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca\") pod \"77f51d6e-439d-4a08-be58-32a14c67cac2\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") "
Nov 03 00:49:14.769799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.769742    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726\") pod \"77f51d6e-439d-4a08-be58-32a14c67cac2\" (UID: \"77f51d6e-439d-4a08-be58-32a14c67cac2\") "
Nov 03 00:49:14.769990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.769964    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca" pod "77f51d6e-439d-4a08-be58-32a14c67cac2" (UID: "77f51d6e-439d-4a08-be58-32a14c67cac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:14.770444 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.770404    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:14.771883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.771855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca" pod "77f51d6e-439d-4a08-be58-32a14c67cac2" (UID: "77f51d6e-439d-4a08-be58-32a14c67cac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:49:14.771975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.771854    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726" pod "77f51d6e-439d-4a08-be58-32a14c67cac2" (UID: "77f51d6e-439d-4a08-be58-32a14c67cac2"). InnerVolumeSpecName "kube-api-access-6l726". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:14.872096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.872059    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f51d6e-439d-4a08-be58-32a14c67cac2-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:14.872096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:14.872090    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l726\" (UniqueName: \"kubernetes.io/projected/77f51d6e-439d-4a08-be58-32a14c67cac2-kube-api-access-6l726\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:15.426250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.426194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:15.526563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.526492    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"
Nov 03 00:49:15.549451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.541477    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6" event={"ID":"77f51d6e-439d-4a08-be58-32a14c67cac2","Type":"ContainerDied","Data":"b3a37a2f2188b34780edc3dbf720a497cd5ef15784e42a9eff5ba4aa2ef1e65d"}
Nov 03 00:49:15.549451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.541564    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3a37a2f2188b34780edc3dbf720a497cd5ef15784e42a9eff5ba4aa2ef1e65d"
Nov 03 00:49:15.549451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.541610    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:15.549451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.543176    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:15.822213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:15.822165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:16.093140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:16.093049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:16.523667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:16.523634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:18.012499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:18.012461    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:18.192679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:18.192641    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:19.920156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:19.920119    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:20.527021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.526978    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw" podUID="4ab7bcac-76bf-4e00-a320-75f5831b69a3" containerName="payload-processing" containerID="cri-o://47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505" gracePeriod=25
Nov 03 00:49:20.821884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.821837    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:20.931043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.931009    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume\") pod \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") "
Nov 03 00:49:20.931598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.931112    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume\") pod \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") "
Nov 03 00:49:20.931598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.931187    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials\") pod \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") "
Nov 03 00:49:20.931598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.931291    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4\") pod \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\" (UID: \"4ab7bcac-76bf-4e00-a320-75f5831b69a3\") "
Nov 03 00:49:20.931598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.931480    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume" pod "4ab7bcac-76bf-4e00-a320-75f5831b69a3" (UID: "4ab7bcac-76bf-4e00-a320-75f5831b69a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:20.931598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.931521    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume" pod "4ab7bcac-76bf-4e00-a320-75f5831b69a3" (UID: "4ab7bcac-76bf-4e00-a320-75f5831b69a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:20.932021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.932004    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:20.932082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.932026    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ab7bcac-76bf-4e00-a320-75f5831b69a3-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:20.933515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.933493    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials" pod "4ab7bcac-76bf-4e00-a320-75f5831b69a3" (UID: "4ab7bcac-76bf-4e00-a320-75f5831b69a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:20.933683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:20.933667    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4" pod "4ab7bcac-76bf-4e00-a320-75f5831b69a3" (UID: "4ab7bcac-76bf-4e00-a320-75f5831b69a3"). InnerVolumeSpecName "kube-api-access-8c5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:21.033965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.033927    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5h4\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-kube-api-access-8c5h4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.033965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.033961    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ab7bcac-76bf-4e00-a320-75f5831b69a3-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.571499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.571459    4550 generic.go:356] "Generic (PLEG): container finished" podID="4ab7bcac-76bf-4e00-a320-75f5831b69a3" containerID="47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505" exitCode=0
Nov 03 00:49:21.571695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.571512    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw" event={"ID":"4ab7bcac-76bf-4e00-a320-75f5831b69a3","Type":"ContainerDied","Data":"47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505"}
Nov 03 00:49:21.571695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.571528    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"
Nov 03 00:49:21.571695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.571581    4550 scope.go:122] "RemoveContainer" containerID="47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505"
Nov 03 00:49:21.571857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.571782    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw" event={"ID":"4ab7bcac-76bf-4e00-a320-75f5831b69a3","Type":"ContainerDied","Data":"094bfa2e7b6d5939a7913512806d850df44d132e9be204802ba182546a3113b3"}
Nov 03 00:49:21.582948 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.582925    4550 scope.go:122] "RemoveContainer" containerID="47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505"
Nov 03 00:49:21.583265 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:49:21.583247    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505\": container with ID starting with 47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505 not found: ID does not exist" containerID="47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505"
Nov 03 00:49:21.583319 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.583282    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505"} err="failed to get container status \"47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505\": rpc error: code = NotFound desc = could not find container \"47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505\": container with ID starting with 47c6af264981d271c27074211af7d276dac4890ffe3c44c78b40cbc7a63f5505 not found: ID does not exist"
Nov 03 00:49:21.628050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.628021    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"]
Nov 03 00:49:21.631774 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.631749    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbb4t5jw"]
Nov 03 00:49:21.658262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.658230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:21.812033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:21.811994    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab7bcac-76bf-4e00-a320-75f5831b69a3" path="/var/lib/kubelet/pods/4ab7bcac-76bf-4e00-a320-75f5831b69a3/volumes"
Nov 03 00:49:22.193755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:22.193710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:22.296796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:22.296752    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:22.397179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:22.397144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:22.554527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:22.552673    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:22.649826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:22.649792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:22.862114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:22.862016    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:23.510721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:23.510684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:24.053955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:24.053914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:24.155094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:24.155047    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:24.725092 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:49:24.725054    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:49:25.216106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.216067    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:25.433242 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.433210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:25.490500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.490409    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78576756d4-8czpf"]
Nov 03 00:49:25.491048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.491033    4550 cpu_manager.go:414] "containerMap: removing container" podUID="77f51d6e-439d-4a08-be58-32a14c67cac2" containerName="revoke-keys"
Nov 03 00:49:25.491097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.491050    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f51d6e-439d-4a08-be58-32a14c67cac2" containerName="revoke-keys"
Nov 03 00:49:25.491097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.491061    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4ab7bcac-76bf-4e00-a320-75f5831b69a3" containerName="payload-processing"
Nov 03 00:49:25.491097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.491066    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab7bcac-76bf-4e00-a320-75f5831b69a3" containerName="payload-processing"
Nov 03 00:49:25.491189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.491166    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f51d6e-439d-4a08-be58-32a14c67cac2" containerName="revoke-keys"
Nov 03 00:49:25.491189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.491175    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab7bcac-76bf-4e00-a320-75f5831b69a3" containerName="payload-processing"
Nov 03 00:49:25.496029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.496006    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.512371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.512347    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78576756d4-8czpf"]
Nov 03 00:49:25.521172 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.521111    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-61d262258feb\""
Nov 03 00:49:25.521377 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.521359    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-psmj8\""
Nov 03 00:49:25.521496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.521374    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:49:25.521496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.521457    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:49:25.525090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.525070    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:49:25.525169 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.525083    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:49:25.745513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.724945    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.763135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.763102    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.763517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.763497    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.763754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.763735    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqmf\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.865801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.865760    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.865801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.865803    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.866082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.865855    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.866082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.865891    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqmf\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.866694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.866667    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.866839 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.866817    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.868505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.868485    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:25.887086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:25.887056    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqmf\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf\") pod \"payload-processing-external-model-78576756d4-8czpf\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") " pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:26.109793 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:26.109709    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:26.412315 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:49:26.412283    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa03907d_d097_4799_995a_025582e55dca.slice/crio-76618f68f2f7115471f12103f0b4654c0b0f3541ea979e568f9f6f1e5fb09b72 WatchSource:0}: Error finding container 76618f68f2f7115471f12103f0b4654c0b0f3541ea979e568f9f6f1e5fb09b72: Status 404 returned error can't find the container with id 76618f68f2f7115471f12103f0b4654c0b0f3541ea979e568f9f6f1e5fb09b72
Nov 03 00:49:26.413623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:26.413595    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78576756d4-8czpf"]
Nov 03 00:49:26.780023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:26.779914    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf" event={"ID":"aa03907d-d097-4799-995a-025582e55dca","Type":"ContainerStarted","Data":"a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a"}
Nov 03 00:49:26.780023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:26.779983    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf" event={"ID":"aa03907d-d097-4799-995a-025582e55dca","Type":"ContainerStarted","Data":"76618f68f2f7115471f12103f0b4654c0b0f3541ea979e568f9f6f1e5fb09b72"}
Nov 03 00:49:26.780023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:26.780015    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:26.822938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:26.822878    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf" podStartSLOduration=1.822863063 podStartE2EDuration="1.822863063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:26.820666622 +0000 UTC m=+2649.142203419" watchObservedRunningTime="2026-11-03 00:49:26.822863063 +0000 UTC m=+2649.144399861"
Nov 03 00:49:27.236847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:27.236807    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:28.655771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:28.655735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:28.756661 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:28.756626    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:29.009980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:29.009888    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:29.615068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:29.614962    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:29.701873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:29.701838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:31.116677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:31.116636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:32.918528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:32.918492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:34.613745 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:34.613714    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:36.242882 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:36.242850    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:37.790609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:37.790578    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:37.830841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:37.830810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:38.479695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.479659    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"]
Nov 03 00:49:38.485463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.485417    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.491307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.491273    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.491611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.491590    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgcd\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.491803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.491777    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.491921 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.491878    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.491983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.491925    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.492048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.491991    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.492048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.492017    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.492168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.492104    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.492168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.492133    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.512229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.512060    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"]
Nov 03 00:49:38.593382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593347    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593397    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593481    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593534    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgcd\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593667    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593715    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593781    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593808    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.593926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593897    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.594096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.593983    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.594212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.594190    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.594402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.594227    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.594402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.594378    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.595909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.595886    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.596677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.596658    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.614089 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.614068    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.614253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.614235    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgcd\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd\") pod \"maas-default-gateway-openshift-default-6cc458d887-6q926\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:38.807320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:38.807239    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:39.123036 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:49:39.123002    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea9f4edf_fbac_4a91_b9a1_0dc3b7caa9b8.slice/crio-9f15453b9fac85be1f2117e030eb325744d3fc72f6fc7577bc81453dfcf35bed WatchSource:0}: Error finding container 9f15453b9fac85be1f2117e030eb325744d3fc72f6fc7577bc81453dfcf35bed: Status 404 returned error can't find the container with id 9f15453b9fac85be1f2117e030eb325744d3fc72f6fc7577bc81453dfcf35bed
Nov 03 00:49:39.125153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.125129    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"]
Nov 03 00:49:39.125903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.125875    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:39.125957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.125939    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:39.125998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.125966    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:49:39.536071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.536034    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:39.916069 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.916027    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" event={"ID":"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8","Type":"ContainerStarted","Data":"3818e4827cb855145d117c946b4f82041cc7c8fc3606d44d311802784b812f3f"}
Nov 03 00:49:39.916585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:39.916091    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" event={"ID":"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8","Type":"ContainerStarted","Data":"9f15453b9fac85be1f2117e030eb325744d3fc72f6fc7577bc81453dfcf35bed"}
Nov 03 00:49:40.808024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:40.807975    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:40.809721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:40.809691    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:40.809878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:40.809746    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:41.294639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:41.294610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:41.807919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:41.807873    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:41.808091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:41.807944    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:42.808138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:42.808101    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:42.808573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:42.808163    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:43.418703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:43.418672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:43.421980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:43.420566    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podStartSLOduration=5.420547022 podStartE2EDuration="5.420547022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:39.985170592 +0000 UTC m=+2662.306707411" watchObservedRunningTime="2026-11-03 00:49:43.420547022 +0000 UTC m=+2665.742083917"
Nov 03 00:49:43.421980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:43.421219    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78576756d4-8czpf"]
Nov 03 00:49:43.808509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:43.808356    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:43.808509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:43.808421    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:44.808111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:44.808077    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:44.808354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:44.808129    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:45.497227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:45.497187    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:45.813596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:45.813502    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:45.813869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:45.813838    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:46.808363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:46.808319    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-6q926 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Nov 03 00:49:46.808919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:46.808424    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Nov 03 00:49:47.240622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:47.240581    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:48.342729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.342693    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:48.343236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.343184    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:48.345690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.345664    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:49:48.421979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.421792    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf" podUID="aa03907d-d097-4799-995a-025582e55dca" containerName="payload-processing" containerID="cri-o://a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a" gracePeriod=25
Nov 03 00:49:48.723732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.721336    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"]
Nov 03 00:49:48.730504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.727182    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.741729 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.737145    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"]
Nov 03 00:49:48.761138 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.761101    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:49:48.761584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.761531    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:49:48.819301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.818730    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.819301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.818995    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zhs\" (UniqueName: \"kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.819301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.819055    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.875052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.874708    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:48.929122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.927840    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf\") pod \"aa03907d-d097-4799-995a-025582e55dca\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") "
Nov 03 00:49:48.929122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.927914    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume\") pod \"aa03907d-d097-4799-995a-025582e55dca\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") "
Nov 03 00:49:48.929122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.927976    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume\") pod \"aa03907d-d097-4799-995a-025582e55dca\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") "
Nov 03 00:49:48.929122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.928012    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials\") pod \"aa03907d-d097-4799-995a-025582e55dca\" (UID: \"aa03907d-d097-4799-995a-025582e55dca\") "
Nov 03 00:49:48.929122 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.928390    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume" pod "aa03907d-d097-4799-995a-025582e55dca" (UID: "aa03907d-d097-4799-995a-025582e55dca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:48.930299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.930238    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume" pod "aa03907d-d097-4799-995a-025582e55dca" (UID: "aa03907d-d097-4799-995a-025582e55dca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:48.932613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.931265    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zhs\" (UniqueName: \"kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.932613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.931338    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.932613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.931635    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.932613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.931791    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:48.932613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.931806    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa03907d-d097-4799-995a-025582e55dca-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:48.932957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.932699    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.966202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.966156    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf" pod "aa03907d-d097-4799-995a-025582e55dca" (UID: "aa03907d-d097-4799-995a-025582e55dca"). InnerVolumeSpecName "kube-api-access-rxqmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:48.967347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.967318    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.968149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.968121    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zhs\" (UniqueName: \"kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs\") pod \"maas-api-revoke-sub-4442b5e0-pkdpk\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:48.969391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:48.969364    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials" pod "aa03907d-d097-4799-995a-025582e55dca" (UID: "aa03907d-d097-4799-995a-025582e55dca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:49.034477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.033323    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:49.034477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.033359    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxqmf\" (UniqueName: \"kubernetes.io/projected/aa03907d-d097-4799-995a-025582e55dca-kube-api-access-rxqmf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:49.052994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.052960    4550 generic.go:356] "Generic (PLEG): container finished" podID="aa03907d-d097-4799-995a-025582e55dca" containerID="a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a" exitCode=0
Nov 03 00:49:49.055385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.053021    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf"
Nov 03 00:49:49.055385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.053044    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf" event={"ID":"aa03907d-d097-4799-995a-025582e55dca","Type":"ContainerDied","Data":"a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a"}
Nov 03 00:49:49.055385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.053368    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78576756d4-8czpf" event={"ID":"aa03907d-d097-4799-995a-025582e55dca","Type":"ContainerDied","Data":"76618f68f2f7115471f12103f0b4654c0b0f3541ea979e568f9f6f1e5fb09b72"}
Nov 03 00:49:49.055385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.053399    4550 scope.go:122] "RemoveContainer" containerID="a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a"
Nov 03 00:49:49.067453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.067398    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:49.068986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.068928    4550 scope.go:122] "RemoveContainer" containerID="a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a"
Nov 03 00:49:49.069347 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:49:49.069299    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a\": container with ID starting with a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a not found: ID does not exist" containerID="a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a"
Nov 03 00:49:49.069482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.069338    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a"} err="failed to get container status \"a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a\": rpc error: code = NotFound desc = could not find container \"a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a\": container with ID starting with a861aa4696aaee70d51dcb53512313fca2250a5b3f168fc8ed9667add71f462a not found: ID does not exist"
Nov 03 00:49:49.120514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.119866    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78576756d4-8czpf"]
Nov 03 00:49:49.126775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.126738    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78576756d4-8czpf"]
Nov 03 00:49:49.164781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.163971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:49.398300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.396410    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"]
Nov 03 00:49:49.811461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:49.811413    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa03907d-d097-4799-995a-025582e55dca" path="/var/lib/kubelet/pods/aa03907d-d097-4799-995a-025582e55dca/volumes"
Nov 03 00:49:50.070565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:50.070465    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk" event={"ID":"dad5aa74-dcd3-453d-b079-b5e1aefcb920","Type":"ContainerStarted","Data":"d46f7b5298d8a94ff72aee7391cf44390e262eed7f83cb419a390a4338a3ca4b"}
Nov 03 00:49:50.070748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:50.070569    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk" event={"ID":"dad5aa74-dcd3-453d-b079-b5e1aefcb920","Type":"ContainerStarted","Data":"674fcc6d8f38248945bfbe8a9bbf317d06f4be432726f026162e725baefebd2b"}
Nov 03 00:49:50.118667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:50.118594    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk" podStartSLOduration=2.118575046 podStartE2EDuration="2.118575046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:50.111741723 +0000 UTC m=+2672.433278531" watchObservedRunningTime="2026-11-03 00:49:50.118575046 +0000 UTC m=+2672.440111846"
Nov 03 00:49:50.924979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:50.924946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:51.008975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:51.008933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:51.161208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:51.161169    4550 generic.go:356] "Generic (PLEG): container finished" podID="dad5aa74-dcd3-453d-b079-b5e1aefcb920" containerID="d46f7b5298d8a94ff72aee7391cf44390e262eed7f83cb419a390a4338a3ca4b" exitCode=0
Nov 03 00:49:51.161561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:51.161536    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk" event={"ID":"dad5aa74-dcd3-453d-b079-b5e1aefcb920","Type":"ContainerDied","Data":"d46f7b5298d8a94ff72aee7391cf44390e262eed7f83cb419a390a4338a3ca4b"}
Nov 03 00:49:52.351764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.351739    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:52.363134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.363093    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca\") pod \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") "
Nov 03 00:49:52.363296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.363222    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs\" (UniqueName: \"kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs\") pod \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") "
Nov 03 00:49:52.363375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.363328    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca\") pod \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\" (UID: \"dad5aa74-dcd3-453d-b079-b5e1aefcb920\") "
Nov 03 00:49:52.364125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.364098    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca" pod "dad5aa74-dcd3-453d-b079-b5e1aefcb920" (UID: "dad5aa74-dcd3-453d-b079-b5e1aefcb920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:52.365950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.365927    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs" pod "dad5aa74-dcd3-453d-b079-b5e1aefcb920" (UID: "dad5aa74-dcd3-453d-b079-b5e1aefcb920"). InnerVolumeSpecName "kube-api-access-56zhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:52.366097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.366073    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca" pod "dad5aa74-dcd3-453d-b079-b5e1aefcb920" (UID: "dad5aa74-dcd3-453d-b079-b5e1aefcb920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:49:52.464868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.464774    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zhs\" (UniqueName: \"kubernetes.io/projected/dad5aa74-dcd3-453d-b079-b5e1aefcb920-kube-api-access-56zhs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:52.464868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.464808    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:52.464868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.464819    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dad5aa74-dcd3-453d-b079-b5e1aefcb920-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:49:52.776465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:52.776366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:53.178794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:53.178764    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"
Nov 03 00:49:53.179022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:53.178765    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk" event={"ID":"dad5aa74-dcd3-453d-b079-b5e1aefcb920","Type":"ContainerDied","Data":"674fcc6d8f38248945bfbe8a9bbf317d06f4be432726f026162e725baefebd2b"}
Nov 03 00:49:53.179022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:53.178888    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="674fcc6d8f38248945bfbe8a9bbf317d06f4be432726f026162e725baefebd2b"
Nov 03 00:49:54.511852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:54.511808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:54.724652 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:49:54.724617    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:49:56.200867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:56.200831    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:57.817607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:57.817575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:59.517391 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:59.517346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:49:59.941733 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:49:59.934018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:00.033974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:00.032014    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:01.525567 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:01.525529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:02.100204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:02.100171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:02.188710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:02.188625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:03.553565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.553534    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"]
Nov 03 00:50:03.554293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.554268    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dad5aa74-dcd3-453d-b079-b5e1aefcb920" containerName="revoke-keys"
Nov 03 00:50:03.554293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.554293    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad5aa74-dcd3-453d-b079-b5e1aefcb920" containerName="revoke-keys"
Nov 03 00:50:03.554478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.554335    4550 cpu_manager.go:414] "containerMap: removing container" podUID="aa03907d-d097-4799-995a-025582e55dca" containerName="payload-processing"
Nov 03 00:50:03.554478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.554341    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa03907d-d097-4799-995a-025582e55dca" containerName="payload-processing"
Nov 03 00:50:03.554478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.554455    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa03907d-d097-4799-995a-025582e55dca" containerName="payload-processing"
Nov 03 00:50:03.554478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.554472    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad5aa74-dcd3-453d-b079-b5e1aefcb920" containerName="revoke-keys"
Nov 03 00:50:03.559984 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.559833    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.576766 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.576735    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b86acbfb246d\""
Nov 03 00:50:03.578027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.577167    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-99029648dbea\""
Nov 03 00:50:03.584153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.584123    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"]
Nov 03 00:50:03.595393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.588096    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-psmj8\""
Nov 03 00:50:03.595393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.589028    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:50:03.595393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.589229    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:50:03.595393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.589377    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:50:03.595393 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.589552    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:50:03.617937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.617905    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:03.670624 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.670547    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.670853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.670729    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.670853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.670768    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885fb\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.673697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.670917    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.772402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.772363    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.772627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.772470    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.772627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.772500    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885fb\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.772627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.772615    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.773293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.773270    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.773419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.773382    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.775522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.775478    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.792447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.792402    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-885fb\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb\") pod \"payload-processing-external-model-5db79bd7f6-kntx2\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:03.896276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:03.896196    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:04.205477 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:50:04.197281    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916f6e5c_6cbe_4eb4_bb55_5fe6eecf7795.slice/crio-23c26f1ab30d2609a6e4a38eff41b35337633ec178119ad45e4ebb0ee77268b6 WatchSource:0}: Error finding container 23c26f1ab30d2609a6e4a38eff41b35337633ec178119ad45e4ebb0ee77268b6: Status 404 returned error can't find the container with id 23c26f1ab30d2609a6e4a38eff41b35337633ec178119ad45e4ebb0ee77268b6
Nov 03 00:50:04.205477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:04.199809    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"]
Nov 03 00:50:04.266236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:04.266205    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2" event={"ID":"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795","Type":"ContainerStarted","Data":"73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51"}
Nov 03 00:50:04.266372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:04.266256    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2" event={"ID":"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795","Type":"ContainerStarted","Data":"23c26f1ab30d2609a6e4a38eff41b35337633ec178119ad45e4ebb0ee77268b6"}
Nov 03 00:50:04.266372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:04.266306    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:04.308585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:04.308535    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2" podStartSLOduration=1.308520606 podStartE2EDuration="1.308520606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:04.305858835 +0000 UTC m=+2686.627395633" watchObservedRunningTime="2026-11-03 00:50:04.308520606 +0000 UTC m=+2686.630057403"
Nov 03 00:50:05.415548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:05.415515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:06.974791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:06.974746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:08.414975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.414942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:08.536135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.536098    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:08.656232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.656046    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"]
Nov 03 00:50:08.663713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.663687    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.694216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.689227    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"]
Nov 03 00:50:08.701550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.700729    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"]
Nov 03 00:50:08.701550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.700857    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.707055 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.707025    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"]
Nov 03 00:50:08.762489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.762457    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"]
Nov 03 00:50:08.850856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.850820    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f174f791-f438-4076-a30c-ecaca62452e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.850856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.850860    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.850882    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.850977    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851050    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851077    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851107    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851190    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851255    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhjb5\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851368    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851413    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851443    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851517    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f174f791-f438-4076-a30c-ecaca62452e5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851611 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851537    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.851725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851605    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f174f791-f438-4076-a30c-ecaca62452e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851661    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851787    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5h96\" (UniqueName: \"kubernetes.io/projected/f174f791-f438-4076-a30c-ecaca62452e5-kube-api-access-q5h96\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.851878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.851824    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.960975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.960895    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.961367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.960452    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.961568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.961551    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.962076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.962057    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.962240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.962223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f174f791-f438-4076-a30c-ecaca62452e5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.962371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.962356    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.962507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.962491    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f174f791-f438-4076-a30c-ecaca62452e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.975450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.962632    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.975840 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.975813    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5h96\" (UniqueName: \"kubernetes.io/projected/f174f791-f438-4076-a30c-ecaca62452e5-kube-api-access-q5h96\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.975970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.975957    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.976171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976158    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f174f791-f438-4076-a30c-ecaca62452e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.976272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976260    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.976366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976354    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.976511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976491    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.976668 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976652    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.976787 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976773    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.976901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976889    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.977012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.976999    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.977115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.977102    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhjb5\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.981447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.962875    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.985509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.978755    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:08.985666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.963795    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f174f791-f438-4076-a30c-ecaca62452e5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.989460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.979214    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:08.989697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.964311    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.979848    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.980387    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.980614    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.981005    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.992976    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f174f791-f438-4076-a30c-ecaca62452e5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:08.998232    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.004048    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f174f791-f438-4076-a30c-ecaca62452e5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.013455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.012172    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5h96\" (UniqueName: \"kubernetes.io/projected/f174f791-f438-4076-a30c-ecaca62452e5-kube-api-access-q5h96\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.014041 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.013716    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f174f791-f438-4076-a30c-ecaca62452e5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-44ncj\" (UID: \"f174f791-f438-4076-a30c-ecaca62452e5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.017454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.016857    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhjb5\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.030455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.027505    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.037498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.034126    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-r2qqg\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.114833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.114667    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:09.296226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.296145    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:09.321663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.321624    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:09.510786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.510746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:09.606576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.606413    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:09.632201 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:50:09.632171    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf174f791_f438_4076_a30c_ecaca62452e5.slice/crio-28af7da7ba823636bd8c8aa1ff56bc5a9003a4cb835f3d6462f1b38e611834e8 WatchSource:0}: Error finding container 28af7da7ba823636bd8c8aa1ff56bc5a9003a4cb835f3d6462f1b38e611834e8: Status 404 returned error can't find the container with id 28af7da7ba823636bd8c8aa1ff56bc5a9003a4cb835f3d6462f1b38e611834e8
Nov 03 00:50:09.632776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.632746    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"]
Nov 03 00:50:09.636267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.636225    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.636359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.636317    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.636399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.636365    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.674959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.674873    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"]
Nov 03 00:50:09.681757 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:50:09.681722    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd14d222_9095_4974_8826_421888f040cf.slice/crio-0193aa8acf176dd493b4e43c11ffaa8f0bddbe5eb768bd1a802c74246e483cfb WatchSource:0}: Error finding container 0193aa8acf176dd493b4e43c11ffaa8f0bddbe5eb768bd1a802c74246e483cfb: Status 404 returned error can't find the container with id 0193aa8acf176dd493b4e43c11ffaa8f0bddbe5eb768bd1a802c74246e483cfb
Nov 03 00:50:09.686019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.685980    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.686128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.686056    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:09.686128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:09.686093    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:50:10.322452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.321582    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" event={"ID":"cd14d222-9095-4974-8826-421888f040cf","Type":"ContainerStarted","Data":"6c99d79172f830c5db450aa4bf5ac00958afef0731f04721f5eb6009681c6dcc"}
Nov 03 00:50:10.322452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.321657    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" event={"ID":"cd14d222-9095-4974-8826-421888f040cf","Type":"ContainerStarted","Data":"0193aa8acf176dd493b4e43c11ffaa8f0bddbe5eb768bd1a802c74246e483cfb"}
Nov 03 00:50:10.322736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.322491    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:10.324304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.323683    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" event={"ID":"f174f791-f438-4076-a30c-ecaca62452e5","Type":"ContainerStarted","Data":"44d9ce2ced616ea16afc213ad8da720affb86e88647a64b4d9bf1b555ccaa85d"}
Nov 03 00:50:10.324304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.323696    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:10.324304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.323725    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" event={"ID":"f174f791-f438-4076-a30c-ecaca62452e5","Type":"ContainerStarted","Data":"28af7da7ba823636bd8c8aa1ff56bc5a9003a4cb835f3d6462f1b38e611834e8"}
Nov 03 00:50:10.324304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.323741    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:10.370697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.370632    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podStartSLOduration=2.3706134309999998 podStartE2EDuration="2.370613431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:10.367352422 +0000 UTC m=+2692.688889212" watchObservedRunningTime="2026-11-03 00:50:10.370613431 +0000 UTC m=+2692.692150231"
Nov 03 00:50:10.412134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:10.412061    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podStartSLOduration=2.412039033 podStartE2EDuration="2.412039033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:10.409612137 +0000 UTC m=+2692.731148954" watchObservedRunningTime="2026-11-03 00:50:10.412039033 +0000 UTC m=+2692.733575832"
Nov 03 00:50:11.299340 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:11.299307    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:11.300091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:11.300068    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:11.300273 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:11.300251    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:11.322603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:11.322572    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:11.322819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:11.322626    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:11.349682 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:11.349649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:12.297455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:12.296866    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:12.297455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:12.296930    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:12.323473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:12.322573    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:12.323473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:12.322642    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:12.606527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:12.604968    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:13.054566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:13.054531    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:13.305605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:13.298851    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:13.305605 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:13.298913    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:13.322384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:13.322345    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:13.322594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:13.322406    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:13.763748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:13.763518    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2" podUID="916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" containerName="payload-processing" containerID="cri-o://73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51" gracePeriod=25
Nov 03 00:50:14.311513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.310931    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:14.311513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.310994    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:14.325300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.324945    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:14.325300 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.325006    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:14.331892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.331854    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:14.427828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.427674    4550 generic.go:356] "Generic (PLEG): container finished" podID="916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" containerID="73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51" exitCode=0
Nov 03 00:50:14.428608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.428322    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2" event={"ID":"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795","Type":"ContainerDied","Data":"73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51"}
Nov 03 00:50:14.428608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.428391    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2" event={"ID":"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795","Type":"ContainerDied","Data":"23c26f1ab30d2609a6e4a38eff41b35337633ec178119ad45e4ebb0ee77268b6"}
Nov 03 00:50:14.428608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.428446    4550 scope.go:122] "RemoveContainer" containerID="73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51"
Nov 03 00:50:14.428608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.428565    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"
Nov 03 00:50:14.435472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.435055    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb\") pod \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") "
Nov 03 00:50:14.435472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.435158    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume\") pod \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") "
Nov 03 00:50:14.435472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.435200    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume\") pod \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") "
Nov 03 00:50:14.436899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.435810    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials\") pod \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\" (UID: \"916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795\") "
Nov 03 00:50:14.437991 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.437966    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume" pod "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" (UID: "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:14.442485 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.441637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume" pod "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" (UID: "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:14.446775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.446757    4550 scope.go:122] "RemoveContainer" containerID="73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51"
Nov 03 00:50:14.447138 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:50:14.447116    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51\": container with ID starting with 73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51 not found: ID does not exist" containerID="73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51"
Nov 03 00:50:14.447218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.447145    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51"} err="failed to get container status \"73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51\": rpc error: code = NotFound desc = could not find container \"73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51\": container with ID starting with 73a1827e8d15d497e481e23e9c98fed5918fbbeaa0a8341b56c2224d3b5b4b51 not found: ID does not exist"
Nov 03 00:50:14.452977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.452954    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb" pod "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" (UID: "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795"). InnerVolumeSpecName "kube-api-access-885fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:14.453246 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.453222    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials" pod "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" (UID: "916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:14.539960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.539867    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:14.539960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.539901    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-885fb\" (UniqueName: \"kubernetes.io/projected/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-kube-api-access-885fb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:14.539960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.539917    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:14.539960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.539931    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:14.821582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.820492    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"]
Nov 03 00:50:14.825329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.825294    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-kntx2"]
Nov 03 00:50:14.826747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:14.826724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:15.314456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:15.298861    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:15.314456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:15.298934    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:15.322334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:15.322300    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:15.322527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:15.322354    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:15.829462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:15.828753    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" path="/var/lib/kubelet/pods/916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795/volumes"
Nov 03 00:50:16.149279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.149139    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:16.299559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.298912    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:16.299851 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.299817    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:16.301275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.301252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:16.328018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.327985    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:16.328313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.328287    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:16.754457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:16.751510    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:17.296588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:17.296550    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:17.297121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:17.296613    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:17.323030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:17.322994    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:17.323306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:17.323283    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:17.823909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:17.823872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:17.939276 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:17.939244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:18.118316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.118222    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:18.296808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.296768    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:50:18.297473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.297395    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused"
Nov 03 00:50:18.322676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.322636    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-r2qqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Nov 03 00:50:18.322883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.322697    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Nov 03 00:50:18.421576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.421527    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:18.751481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:18.751281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:19.051077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:19.050966    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:19.147147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:19.146855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:19.990924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:19.990891    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:19.992881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:19.992862    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:19.997076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:19.997023    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj"
Nov 03 00:50:20.010712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:20.009749    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:20.010992 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:20.010976    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:20.011756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:20.011732    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:50:20.303759 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:20.303723    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:20.912456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:20.911037    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:22.563633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:22.563587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:24.308753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:24.308710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:24.726635 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:50:24.726595    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:50:25.922066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:25.922030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:27.837777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:27.837743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:30.013791 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:30.013739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:31.842383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:31.842340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:32.731025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:32.730986    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:33.704299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:33.704270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:35.400020 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:35.399984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:37.130598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:37.130563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:38.830971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:38.830933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:39.289490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.289448    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"]
Nov 03 00:50:39.290470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.290419    4550 cpu_manager.go:414] "containerMap: removing container" podUID="916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" containerName="payload-processing"
Nov 03 00:50:39.290470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.290456    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" containerName="payload-processing"
Nov 03 00:50:39.290691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.290648    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="916f6e5c-6cbe-4eb4-bb55-5fe6eecf7795" containerName="payload-processing"
Nov 03 00:50:39.296493 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.296463    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.311731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.311703    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"]
Nov 03 00:50:39.317914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.317891    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:50:39.318066 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.317986    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:50:39.368383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.368331    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmwr\" (UniqueName: \"kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.368383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.368385    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.368670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.368544    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.469892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.469856    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmwr\" (UniqueName: \"kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.470170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.470150    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.470356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.470340    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.471962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.471936    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.474854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.474831    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.491257 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.491227    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmwr\" (UniqueName: \"kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr\") pod \"maas-api-revoke-sub-f7a50f76-lf7fj\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.616098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.616018    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:39.921524 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:50:39.921480    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5246569a_89e7_4734_9b33_27b51f274531.slice/crio-01cf4b648d49d1c26de6aeeb72520aeb726f4537eaf4cd9ebd5c2acc15ac2bb9 WatchSource:0}: Error finding container 01cf4b648d49d1c26de6aeeb72520aeb726f4537eaf4cd9ebd5c2acc15ac2bb9: Status 404 returned error can't find the container with id 01cf4b648d49d1c26de6aeeb72520aeb726f4537eaf4cd9ebd5c2acc15ac2bb9
Nov 03 00:50:39.924576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:39.924535    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"]
Nov 03 00:50:40.582311 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:40.582206    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:40.875291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:40.875192    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj" event={"ID":"5246569a-89e7-4734-9b33-27b51f274531","Type":"ContainerStarted","Data":"1278e47955562051a6b84920e8ca483a30749b134930eae64addd5ee19501799"}
Nov 03 00:50:40.875291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:40.875264    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj" event={"ID":"5246569a-89e7-4734-9b33-27b51f274531","Type":"ContainerStarted","Data":"01cf4b648d49d1c26de6aeeb72520aeb726f4537eaf4cd9ebd5c2acc15ac2bb9"}
Nov 03 00:50:40.918578 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:40.918489    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj" podStartSLOduration=1.9184735929999999 podStartE2EDuration="1.918473593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:40.913989603 +0000 UTC m=+2723.235526404" watchObservedRunningTime="2026-11-03 00:50:40.918473593 +0000 UTC m=+2723.240010392"
Nov 03 00:50:41.885483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:41.885412    4550 generic.go:356] "Generic (PLEG): container finished" podID="5246569a-89e7-4734-9b33-27b51f274531" containerID="1278e47955562051a6b84920e8ca483a30749b134930eae64addd5ee19501799" exitCode=0
Nov 03 00:50:41.885929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:41.885501    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj" event={"ID":"5246569a-89e7-4734-9b33-27b51f274531","Type":"ContainerDied","Data":"1278e47955562051a6b84920e8ca483a30749b134930eae64addd5ee19501799"}
Nov 03 00:50:42.300386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:42.300350    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:43.089298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.089268    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:43.238228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.238194    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca\") pod \"5246569a-89e7-4734-9b33-27b51f274531\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") "
Nov 03 00:50:43.238476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.238333    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca\") pod \"5246569a-89e7-4734-9b33-27b51f274531\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") "
Nov 03 00:50:43.238556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.238517    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr\" (UniqueName: \"kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr\") pod \"5246569a-89e7-4734-9b33-27b51f274531\" (UID: \"5246569a-89e7-4734-9b33-27b51f274531\") "
Nov 03 00:50:43.238612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.238563    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca" pod "5246569a-89e7-4734-9b33-27b51f274531" (UID: "5246569a-89e7-4734-9b33-27b51f274531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:43.239088 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.239069    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246569a-89e7-4734-9b33-27b51f274531-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:43.240342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.240319    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca" pod "5246569a-89e7-4734-9b33-27b51f274531" (UID: "5246569a-89e7-4734-9b33-27b51f274531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:50:43.240538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.240518    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr" pod "5246569a-89e7-4734-9b33-27b51f274531" (UID: "5246569a-89e7-4734-9b33-27b51f274531"). InnerVolumeSpecName "kube-api-access-9rmwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:43.341134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.341095    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246569a-89e7-4734-9b33-27b51f274531-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:43.341134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.341133    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rmwr\" (UniqueName: \"kubernetes.io/projected/5246569a-89e7-4734-9b33-27b51f274531-kube-api-access-9rmwr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:43.902006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.901979    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"
Nov 03 00:50:43.902220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.902018    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj" event={"ID":"5246569a-89e7-4734-9b33-27b51f274531","Type":"ContainerDied","Data":"01cf4b648d49d1c26de6aeeb72520aeb726f4537eaf4cd9ebd5c2acc15ac2bb9"}
Nov 03 00:50:43.902220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:43.902076    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01cf4b648d49d1c26de6aeeb72520aeb726f4537eaf4cd9ebd5c2acc15ac2bb9"
Nov 03 00:50:44.043317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:44.043281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:45.759573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:45.759540    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:47.448374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:47.448331    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:49.109599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:49.109537    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:50.632767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:50.632732    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:50.747458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:50.741095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:50.900772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:50.900738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:51.292509 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.291589    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"]
Nov 03 00:50:51.292709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.292630    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5246569a-89e7-4734-9b33-27b51f274531" containerName="revoke-keys"
Nov 03 00:50:51.292709 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.292651    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5246569a-89e7-4734-9b33-27b51f274531" containerName="revoke-keys"
Nov 03 00:50:51.292827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.292797    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5246569a-89e7-4734-9b33-27b51f274531" containerName="revoke-keys"
Nov 03 00:50:51.314540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.297056    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.353619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.351778    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:50:51.353619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.351851    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:50:51.397506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.395050    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"]
Nov 03 00:50:51.398813 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.398785    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9g7\" (UniqueName: \"kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.399581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.399549    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.399890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.399852    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.502052 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.502017    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.502278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.502069    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.502278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.502219    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9g7\" (UniqueName: \"kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.502873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.502846    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.504614 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.504592    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.522098 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.522073    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9g7\" (UniqueName: \"kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7\") pod \"maas-api-revoke-sub-f14627cb-pmwfj\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.691553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.691519    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:51.995956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:51.995928    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"]
Nov 03 00:50:52.726007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:52.725970    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:52.973862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:52.973829    4550 generic.go:356] "Generic (PLEG): container finished" podID="271a04b5-aff2-475f-a4be-2d4e7a34313f" containerID="e8423f1b475e49a9d407b03d04d5906e74bc536dc8a76decb94178a441fe7b0e" exitCode=0
Nov 03 00:50:52.974050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:52.973869    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj" event={"ID":"271a04b5-aff2-475f-a4be-2d4e7a34313f","Type":"ContainerDied","Data":"e8423f1b475e49a9d407b03d04d5906e74bc536dc8a76decb94178a441fe7b0e"}
Nov 03 00:50:52.974050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:52.973905    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj" event={"ID":"271a04b5-aff2-475f-a4be-2d4e7a34313f","Type":"ContainerStarted","Data":"5e56fab138b87f5985c3c8601fa05c398252d51da134f8a9dbaede1bc675b703"}
Nov 03 00:50:54.157109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.157075    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:54.255591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.255482    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca\") pod \"271a04b5-aff2-475f-a4be-2d4e7a34313f\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") "
Nov 03 00:50:54.255794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.255608    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7\" (UniqueName: \"kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7\") pod \"271a04b5-aff2-475f-a4be-2d4e7a34313f\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") "
Nov 03 00:50:54.255794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.255634    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca\") pod \"271a04b5-aff2-475f-a4be-2d4e7a34313f\" (UID: \"271a04b5-aff2-475f-a4be-2d4e7a34313f\") "
Nov 03 00:50:54.255962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.255933    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca" pod "271a04b5-aff2-475f-a4be-2d4e7a34313f" (UID: "271a04b5-aff2-475f-a4be-2d4e7a34313f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:54.256342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.256320    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:54.258552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.258525    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca" pod "271a04b5-aff2-475f-a4be-2d4e7a34313f" (UID: "271a04b5-aff2-475f-a4be-2d4e7a34313f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:50:54.258645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.258622    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7" pod "271a04b5-aff2-475f-a4be-2d4e7a34313f" (UID: "271a04b5-aff2-475f-a4be-2d4e7a34313f"). InnerVolumeSpecName "kube-api-access-2d9g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:54.357346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.357306    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9g7\" (UniqueName: \"kubernetes.io/projected/271a04b5-aff2-475f-a4be-2d4e7a34313f-kube-api-access-2d9g7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:54.357346 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.357337    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/271a04b5-aff2-475f-a4be-2d4e7a34313f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:50:54.421712 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.421681    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:54.724233 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:50:54.724196    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:50:54.991638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.991529    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"
Nov 03 00:50:54.991638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.991562    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj" event={"ID":"271a04b5-aff2-475f-a4be-2d4e7a34313f","Type":"ContainerDied","Data":"5e56fab138b87f5985c3c8601fa05c398252d51da134f8a9dbaede1bc675b703"}
Nov 03 00:50:54.991638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:54.991613    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e56fab138b87f5985c3c8601fa05c398252d51da134f8a9dbaede1bc675b703"
Nov 03 00:50:56.008203 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:56.008167    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:57.747801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:57.747757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:50:59.516158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:50:59.516079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:01.197551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:01.197507    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:01.725857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:01.725818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:02.104245 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.104154    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"]
Nov 03 00:51:02.104825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.104810    4550 cpu_manager.go:414] "containerMap: removing container" podUID="271a04b5-aff2-475f-a4be-2d4e7a34313f" containerName="revoke-keys"
Nov 03 00:51:02.104874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.104826    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271a04b5-aff2-475f-a4be-2d4e7a34313f" containerName="revoke-keys"
Nov 03 00:51:02.104929 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.104920    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="271a04b5-aff2-475f-a4be-2d4e7a34313f" containerName="revoke-keys"
Nov 03 00:51:02.108726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.108710    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.125944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.125916    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"]
Nov 03 00:51:02.134722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.134698    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:51:02.134876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.134777    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:51:02.187147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.187108    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.187589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.187549    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564k5\" (UniqueName: \"kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.187740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.187722    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.289725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.289683    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.290230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.289775    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564k5\" (UniqueName: \"kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.290230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.289888    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.290371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.290341    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.292261 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.292229    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.310108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.310085    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564k5\" (UniqueName: \"kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5\") pod \"maas-api-revoke-sub-216209e3-xzqp9\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.421926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.421892    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:02.727902 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:51:02.727866    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e617fc_b30c_4df8_ba73_fde2083cbf72.slice/crio-04884e46aa497ca20a3560aca8d0971148be0cd50cb240ab0e7b9687b169f98b WatchSource:0}: Error finding container 04884e46aa497ca20a3560aca8d0971148be0cd50cb240ab0e7b9687b169f98b: Status 404 returned error can't find the container with id 04884e46aa497ca20a3560aca8d0971148be0cd50cb240ab0e7b9687b169f98b
Nov 03 00:51:02.730202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.730180    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"]
Nov 03 00:51:02.936461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:02.936394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:03.060855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:03.060812    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9" event={"ID":"60e617fc-b30c-4df8-ba73-fde2083cbf72","Type":"ContainerStarted","Data":"04884e46aa497ca20a3560aca8d0971148be0cd50cb240ab0e7b9687b169f98b"}
Nov 03 00:51:04.071895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:04.071805    4550 generic.go:356] "Generic (PLEG): container finished" podID="60e617fc-b30c-4df8-ba73-fde2083cbf72" containerID="17ad68fee563a9f6add52e8beb2b433aef6c51f884e3592d4d8a6048939931d6" exitCode=0
Nov 03 00:51:04.071895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:04.071848    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9" event={"ID":"60e617fc-b30c-4df8-ba73-fde2083cbf72","Type":"ContainerDied","Data":"17ad68fee563a9f6add52e8beb2b433aef6c51f884e3592d4d8a6048939931d6"}
Nov 03 00:51:04.534767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:04.534733    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:05.255193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.255165    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:05.344358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.344264    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca\") pod \"60e617fc-b30c-4df8-ba73-fde2083cbf72\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") "
Nov 03 00:51:05.344549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.344371    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5\" (UniqueName: \"kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5\") pod \"60e617fc-b30c-4df8-ba73-fde2083cbf72\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") "
Nov 03 00:51:05.344549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.344479    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca\") pod \"60e617fc-b30c-4df8-ba73-fde2083cbf72\" (UID: \"60e617fc-b30c-4df8-ba73-fde2083cbf72\") "
Nov 03 00:51:05.344760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.344736    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca" pod "60e617fc-b30c-4df8-ba73-fde2083cbf72" (UID: "60e617fc-b30c-4df8-ba73-fde2083cbf72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:05.345235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.345210    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:05.347087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.347063    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5" pod "60e617fc-b30c-4df8-ba73-fde2083cbf72" (UID: "60e617fc-b30c-4df8-ba73-fde2083cbf72"). InnerVolumeSpecName "kube-api-access-564k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:05.347218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.347200    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca" pod "60e617fc-b30c-4df8-ba73-fde2083cbf72" (UID: "60e617fc-b30c-4df8-ba73-fde2083cbf72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:51:05.446491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.446427    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60e617fc-b30c-4df8-ba73-fde2083cbf72-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:05.446491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:05.446489    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564k5\" (UniqueName: \"kubernetes.io/projected/60e617fc-b30c-4df8-ba73-fde2083cbf72-kube-api-access-564k5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:06.089969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:06.089935    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"
Nov 03 00:51:06.089969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:06.089955    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9" event={"ID":"60e617fc-b30c-4df8-ba73-fde2083cbf72","Type":"ContainerDied","Data":"04884e46aa497ca20a3560aca8d0971148be0cd50cb240ab0e7b9687b169f98b"}
Nov 03 00:51:06.090193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:06.090006    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04884e46aa497ca20a3560aca8d0971148be0cd50cb240ab0e7b9687b169f98b"
Nov 03 00:51:06.358007 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:06.357974    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:08.259455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:08.258948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:09.623456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:09.608549    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:51:09.885554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:09.763502    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:51:09.885554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:09.841767    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:51:09.918476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:09.918375    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:51:10.113499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.113461    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:10.211850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.211809    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"]
Nov 03 00:51:10.212724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.212676    4550 cpu_manager.go:414] "containerMap: removing container" podUID="60e617fc-b30c-4df8-ba73-fde2083cbf72" containerName="revoke-keys"
Nov 03 00:51:10.212724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.212698    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e617fc-b30c-4df8-ba73-fde2083cbf72" containerName="revoke-keys"
Nov 03 00:51:10.212940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.212891    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e617fc-b30c-4df8-ba73-fde2083cbf72" containerName="revoke-keys"
Nov 03 00:51:10.217378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.217353    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.238836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.238798    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"]
Nov 03 00:51:10.253484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.249799    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:51:10.253484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.249914    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:51:10.317717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.317670    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.317936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.317727    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.318001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.317948    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7hk\" (UniqueName: \"kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.418867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.418834    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7hk\" (UniqueName: \"kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.419061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.418938    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.419061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.418960    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.419711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.419691    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.421351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.421334    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.438847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.438821    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7hk\" (UniqueName: \"kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk\") pod \"maas-api-revoke-sub-78888f57-248nm\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.538673 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.538584    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:10.849252 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:51:10.849225    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1082eb74_bb4a_4295_bdcc_a3f9dca1b478.slice/crio-be30925429d8cff28a46ff0b1fe0e4c68d9f551cf74db97bae72e42dae8491a6 WatchSource:0}: Error finding container be30925429d8cff28a46ff0b1fe0e4c68d9f551cf74db97bae72e42dae8491a6: Status 404 returned error can't find the container with id be30925429d8cff28a46ff0b1fe0e4c68d9f551cf74db97bae72e42dae8491a6
Nov 03 00:51:10.851147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:10.851117    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"]
Nov 03 00:51:11.271704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:11.271675    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm" event={"ID":"1082eb74-bb4a-4295-bdcc-a3f9dca1b478","Type":"ContainerStarted","Data":"be30925429d8cff28a46ff0b1fe0e4c68d9f551cf74db97bae72e42dae8491a6"}
Nov 03 00:51:11.724768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:11.724729    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:12.281179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:12.281143    4550 generic.go:356] "Generic (PLEG): container finished" podID="1082eb74-bb4a-4295-bdcc-a3f9dca1b478" containerID="ab44ef6c5862e943876e6cea20b46d27bd5e4a9d01451ebe109cca0ef9b17874" exitCode=0
Nov 03 00:51:12.281700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:12.281227    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm" event={"ID":"1082eb74-bb4a-4295-bdcc-a3f9dca1b478","Type":"ContainerDied","Data":"ab44ef6c5862e943876e6cea20b46d27bd5e4a9d01451ebe109cca0ef9b17874"}
Nov 03 00:51:12.698853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:12.698816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:13.226895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.226857    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:13.348291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.344009    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:13.520047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.520001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:13.669105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.669075    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:13.777121 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.777020    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca\") pod \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") "
Nov 03 00:51:13.777408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.777387    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca\") pod \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") "
Nov 03 00:51:13.777537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.777424    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk\" (UniqueName: \"kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk\") pod \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\" (UID: \"1082eb74-bb4a-4295-bdcc-a3f9dca1b478\") "
Nov 03 00:51:13.777785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.777756    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca" pod "1082eb74-bb4a-4295-bdcc-a3f9dca1b478" (UID: "1082eb74-bb4a-4295-bdcc-a3f9dca1b478"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:13.778111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.778089    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:13.779305 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.779276    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca" pod "1082eb74-bb4a-4295-bdcc-a3f9dca1b478" (UID: "1082eb74-bb4a-4295-bdcc-a3f9dca1b478"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:51:13.779742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.779724    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk" pod "1082eb74-bb4a-4295-bdcc-a3f9dca1b478" (UID: "1082eb74-bb4a-4295-bdcc-a3f9dca1b478"). InnerVolumeSpecName "kube-api-access-cd7hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:13.879965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.879929    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd7hk\" (UniqueName: \"kubernetes.io/projected/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-kube-api-access-cd7hk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:13.879965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:13.879962    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082eb74-bb4a-4295-bdcc-a3f9dca1b478-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:14.313691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:14.305112    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm" event={"ID":"1082eb74-bb4a-4295-bdcc-a3f9dca1b478","Type":"ContainerDied","Data":"be30925429d8cff28a46ff0b1fe0e4c68d9f551cf74db97bae72e42dae8491a6"}
Nov 03 00:51:14.313691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:14.305182    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be30925429d8cff28a46ff0b1fe0e4c68d9f551cf74db97bae72e42dae8491a6"
Nov 03 00:51:14.313691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:14.305273    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"
Nov 03 00:51:14.339981 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:14.339941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:14.426197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:14.426161    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:14.949484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:14.949379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:15.476953 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:15.476921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:17.226084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:17.226044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:18.813613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:18.813578    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:20.532955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:20.532915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:20.972785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:20.972743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:21.081933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.081905    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:21.124114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.124077    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"]
Nov 03 00:51:21.125045 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.125025    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1082eb74-bb4a-4295-bdcc-a3f9dca1b478" containerName="revoke-keys"
Nov 03 00:51:21.125111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.125048    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1082eb74-bb4a-4295-bdcc-a3f9dca1b478" containerName="revoke-keys"
Nov 03 00:51:21.125244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.125232    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1082eb74-bb4a-4295-bdcc-a3f9dca1b478" containerName="revoke-keys"
Nov 03 00:51:21.129113 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.129094    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.132092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.132063    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.132225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.132104    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.132225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.132122    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjp4\" (UniqueName: \"kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.147720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.147681    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"]
Nov 03 00:51:21.155603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.155568    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:51:21.155757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.155621    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:51:21.187262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.187225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:21.233667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.233574    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjp4\" (UniqueName: \"kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.233853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.233793    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.233853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.233823    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.234445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.234417    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.236163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.236143    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.254503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.254459    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjp4\" (UniqueName: \"kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4\") pod \"maas-api-revoke-sub-9e2ca183-kq6tm\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.443697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.443660    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:21.744105 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:51:21.744074    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135d6a8a_4dfa_4aca_960e_2c41a158fd7f.slice/crio-ad46a72672c7fcc2fefdd0a89b9fbb6f67f44c668f8e14fcbdff723018ae5340 WatchSource:0}: Error finding container ad46a72672c7fcc2fefdd0a89b9fbb6f67f44c668f8e14fcbdff723018ae5340: Status 404 returned error can't find the container with id ad46a72672c7fcc2fefdd0a89b9fbb6f67f44c668f8e14fcbdff723018ae5340
Nov 03 00:51:21.747852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:21.747832    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"]
Nov 03 00:51:22.335005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:22.334918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:22.373214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:22.373180    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm" event={"ID":"135d6a8a-4dfa-4aca-960e-2c41a158fd7f","Type":"ContainerStarted","Data":"ac67cf724d3112a376f511a64c8d42525e180e870e6a574a64af91857895e987"}
Nov 03 00:51:22.373570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:22.373230    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm" event={"ID":"135d6a8a-4dfa-4aca-960e-2c41a158fd7f","Type":"ContainerStarted","Data":"ad46a72672c7fcc2fefdd0a89b9fbb6f67f44c668f8e14fcbdff723018ae5340"}
Nov 03 00:51:23.384238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:23.384201    4550 generic.go:356] "Generic (PLEG): container finished" podID="135d6a8a-4dfa-4aca-960e-2c41a158fd7f" containerID="ac67cf724d3112a376f511a64c8d42525e180e870e6a574a64af91857895e987" exitCode=0
Nov 03 00:51:23.384763 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:23.384287    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm" event={"ID":"135d6a8a-4dfa-4aca-960e-2c41a158fd7f","Type":"ContainerDied","Data":"ac67cf724d3112a376f511a64c8d42525e180e870e6a574a64af91857895e987"}
Nov 03 00:51:24.000200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.000159    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:24.553291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.553262    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:24.592227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.592188    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4\" (UniqueName: \"kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4\") pod \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") "
Nov 03 00:51:24.592451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.592292    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca\") pod \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") "
Nov 03 00:51:24.592451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.592396    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca\") pod \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\" (UID: \"135d6a8a-4dfa-4aca-960e-2c41a158fd7f\") "
Nov 03 00:51:24.593272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.593066    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca" pod "135d6a8a-4dfa-4aca-960e-2c41a158fd7f" (UID: "135d6a8a-4dfa-4aca-960e-2c41a158fd7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:24.593272 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.593195    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:24.594875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.594844    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4" pod "135d6a8a-4dfa-4aca-960e-2c41a158fd7f" (UID: "135d6a8a-4dfa-4aca-960e-2c41a158fd7f"). InnerVolumeSpecName "kube-api-access-7hjp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:24.595585 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.595562    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca" pod "135d6a8a-4dfa-4aca-960e-2c41a158fd7f" (UID: "135d6a8a-4dfa-4aca-960e-2c41a158fd7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:51:24.694939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.694907    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hjp4\" (UniqueName: \"kubernetes.io/projected/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-kube-api-access-7hjp4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:24.694939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:24.694937    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135d6a8a-4dfa-4aca-960e-2c41a158fd7f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:24.724875 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:51:24.724834    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:51:25.415093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:25.415060    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"
Nov 03 00:51:25.415379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:25.415036    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm" event={"ID":"135d6a8a-4dfa-4aca-960e-2c41a158fd7f","Type":"ContainerDied","Data":"ad46a72672c7fcc2fefdd0a89b9fbb6f67f44c668f8e14fcbdff723018ae5340"}
Nov 03 00:51:25.415379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:25.415228    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad46a72672c7fcc2fefdd0a89b9fbb6f67f44c668f8e14fcbdff723018ae5340"
Nov 03 00:51:25.731146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:25.731050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:27.071554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:27.071518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:27.162503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:27.156381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:27.512163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:27.512131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:28.031458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:28.029208    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:28.127116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:28.127081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:29.517600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:29.517552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:31.146701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:31.146665    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:32.372230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:32.372196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:32.499989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:32.499952    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:33.034946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:33.034905    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:33.480403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:33.480368    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:33.564732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:33.564698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:34.114361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.114326    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"]
Nov 03 00:51:34.115028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.115011    4550 cpu_manager.go:414] "containerMap: removing container" podUID="135d6a8a-4dfa-4aca-960e-2c41a158fd7f" containerName="revoke-keys"
Nov 03 00:51:34.115086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.115029    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135d6a8a-4dfa-4aca-960e-2c41a158fd7f" containerName="revoke-keys"
Nov 03 00:51:34.115141 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.115131    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="135d6a8a-4dfa-4aca-960e-2c41a158fd7f" containerName="revoke-keys"
Nov 03 00:51:34.119038 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.119019    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.138220 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.138192    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"]
Nov 03 00:51:34.146679 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.146647    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:51:34.146829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.146694    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:51:34.227632 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.227602    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mrm\" (UniqueName: \"kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.227827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.227763    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.227827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.227809    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.329502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.329467    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mrm\" (UniqueName: \"kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.329720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.329594    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.329720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.329635    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.330385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.330356    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.332133 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.332101    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.351976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.351936    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mrm\" (UniqueName: \"kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm\") pod \"maas-api-revoke-sub-9a3ac151-wsw2w\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.433251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.433223    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:34.508135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.508103    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:34.606491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.604366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:34.812361 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:34.812329    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"]
Nov 03 00:51:35.070002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:35.069973    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:35.574021 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:35.573982    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w" event={"ID":"d4ace347-b815-421f-9086-144598bbfa34","Type":"ContainerStarted","Data":"5671da8470cdbf82ec09d60540772de4148a757c26a768823a9230ab0c263309"}
Nov 03 00:51:35.574589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:35.574069    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w" event={"ID":"d4ace347-b815-421f-9086-144598bbfa34","Type":"ContainerStarted","Data":"2b55dc877f2a0b9d4b54da3783bcddc5a8a1805e70d5e662e302ac9227510fb5"}
Nov 03 00:51:35.618686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:35.618630    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w" podStartSLOduration=1.6186170199999999 podStartE2EDuration="1.61861702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:35.615191014 +0000 UTC m=+2777.936727813" watchObservedRunningTime="2026-11-03 00:51:35.61861702 +0000 UTC m=+2777.940153817"
Nov 03 00:51:36.585810 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:36.585769    4550 generic.go:356] "Generic (PLEG): container finished" podID="d4ace347-b815-421f-9086-144598bbfa34" containerID="5671da8470cdbf82ec09d60540772de4148a757c26a768823a9230ab0c263309" exitCode=0
Nov 03 00:51:36.586333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:36.585833    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w" event={"ID":"d4ace347-b815-421f-9086-144598bbfa34","Type":"ContainerDied","Data":"5671da8470cdbf82ec09d60540772de4148a757c26a768823a9230ab0c263309"}
Nov 03 00:51:36.925543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:36.925508    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:37.296012 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.295923    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:37.396030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.395993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:37.590083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.589989    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:37.832374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.832347    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:37.882110 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.881884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:37.918816 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.918777    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca\") pod \"d4ace347-b815-421f-9086-144598bbfa34\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") "
Nov 03 00:51:37.919026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.918933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca\") pod \"d4ace347-b815-421f-9086-144598bbfa34\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") "
Nov 03 00:51:37.919026 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.918977    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm\" (UniqueName: \"kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm\") pod \"d4ace347-b815-421f-9086-144598bbfa34\" (UID: \"d4ace347-b815-421f-9086-144598bbfa34\") "
Nov 03 00:51:37.919888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.919852    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca" pod "d4ace347-b815-421f-9086-144598bbfa34" (UID: "d4ace347-b815-421f-9086-144598bbfa34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:37.922221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.922195    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca" pod "d4ace347-b815-421f-9086-144598bbfa34" (UID: "d4ace347-b815-421f-9086-144598bbfa34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:51:37.922875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:37.922842    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm" pod "d4ace347-b815-421f-9086-144598bbfa34" (UID: "d4ace347-b815-421f-9086-144598bbfa34"). InnerVolumeSpecName "kube-api-access-n7mrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:38.021864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.021834    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ace347-b815-421f-9086-144598bbfa34-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:38.021864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.021861    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ace347-b815-421f-9086-144598bbfa34-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:38.021864 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.021872    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mrm\" (UniqueName: \"kubernetes.io/projected/d4ace347-b815-421f-9086-144598bbfa34-kube-api-access-n7mrm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:38.476754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.476673    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:38.567071 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.566943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:38.606217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.606172    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w" event={"ID":"d4ace347-b815-421f-9086-144598bbfa34","Type":"ContainerDied","Data":"2b55dc877f2a0b9d4b54da3783bcddc5a8a1805e70d5e662e302ac9227510fb5"}
Nov 03 00:51:38.606670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.606243    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b55dc877f2a0b9d4b54da3783bcddc5a8a1805e70d5e662e302ac9227510fb5"
Nov 03 00:51:38.606670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.606318    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"
Nov 03 00:51:38.927455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:38.924611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:40.467940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:40.467900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:41.298056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:41.298015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:41.481535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:41.481491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:43.219609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:43.219575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:45.257752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:45.257707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:46.729468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:46.715867    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:46.928386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:46.923387    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:47.083720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:47.083556    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:47.449833 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:47.449798    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:47.979968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:47.979933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:48.393565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:48.392701    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:51:48.438467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:48.437530    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:51:49.736612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:49.736580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:51.410706 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.410668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:51.893773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.893686    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"]
Nov 03 00:51:51.894731 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.894710    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d4ace347-b815-421f-9086-144598bbfa34" containerName="revoke-keys"
Nov 03 00:51:51.894806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.894733    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ace347-b815-421f-9086-144598bbfa34" containerName="revoke-keys"
Nov 03 00:51:51.894901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.894889    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ace347-b815-421f-9086-144598bbfa34" containerName="revoke-keys"
Nov 03 00:51:51.899297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.899265    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:51.915128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.915099    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"]
Nov 03 00:51:51.922710 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.922069    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:51:51.922998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:51.922970    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:51:52.011698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.011667    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.011887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.011787    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.011955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.011930    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbtf\" (UniqueName: \"kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.113586 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.113550    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.113809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.113636    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.113809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.113735    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbtf\" (UniqueName: \"kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.114299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.114267    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.115965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.115941    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.134077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.134042    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbtf\" (UniqueName: \"kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf\") pod \"maas-api-revoke-sub-ea2c7dda-jkkjb\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.217081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.217042    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:52.527201 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:51:52.527170    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9372f1_08ac_4d58_ad29_a0759f1dea35.slice/crio-c05221a1e4625309badbd3e577bb7b564b53048bcd8333f0946552fad3c5bb8a WatchSource:0}: Error finding container c05221a1e4625309badbd3e577bb7b564b53048bcd8333f0946552fad3c5bb8a: Status 404 returned error can't find the container with id c05221a1e4625309badbd3e577bb7b564b53048bcd8333f0946552fad3c5bb8a
Nov 03 00:51:52.529569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:52.529545    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"]
Nov 03 00:51:53.031638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.031555    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb" event={"ID":"7c9372f1-08ac-4d58-ad29-a0759f1dea35","Type":"ContainerStarted","Data":"1e6d6192cc7505a3fa42fb8cc6ce602058b5624e80b2f1286902a0f76ecaf1a7"}
Nov 03 00:51:53.031638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.031606    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb" event={"ID":"7c9372f1-08ac-4d58-ad29-a0759f1dea35","Type":"ContainerStarted","Data":"c05221a1e4625309badbd3e577bb7b564b53048bcd8333f0946552fad3c5bb8a"}
Nov 03 00:51:53.074826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.074755    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb" podStartSLOduration=2.074735266 podStartE2EDuration="2.074735266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:53.071155551 +0000 UTC m=+2795.392692379" watchObservedRunningTime="2026-11-03 00:51:53.074735266 +0000 UTC m=+2795.396272069"
Nov 03 00:51:53.155790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.155753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:53.408284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.408251    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:53.514384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.514346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:53.615197 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:53.615161    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:54.040522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:54.040484    4550 generic.go:356] "Generic (PLEG): container finished" podID="7c9372f1-08ac-4d58-ad29-a0759f1dea35" containerID="1e6d6192cc7505a3fa42fb8cc6ce602058b5624e80b2f1286902a0f76ecaf1a7" exitCode=0
Nov 03 00:51:54.040707 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:54.040563    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb" event={"ID":"7c9372f1-08ac-4d58-ad29-a0759f1dea35","Type":"ContainerDied","Data":"1e6d6192cc7505a3fa42fb8cc6ce602058b5624e80b2f1286902a0f76ecaf1a7"}
Nov 03 00:51:54.724870 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:51:54.724834    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:51:54.963487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:54.963457    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:55.258140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.258109    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:55.379423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.379323    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf\" (UniqueName: \"kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf\") pod \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") "
Nov 03 00:51:55.379655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.379461    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca\") pod \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") "
Nov 03 00:51:55.379655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.379536    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca\") pod \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\" (UID: \"7c9372f1-08ac-4d58-ad29-a0759f1dea35\") "
Nov 03 00:51:55.380383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.380334    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca" pod "7c9372f1-08ac-4d58-ad29-a0759f1dea35" (UID: "7c9372f1-08ac-4d58-ad29-a0759f1dea35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:55.382497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.382473    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca" pod "7c9372f1-08ac-4d58-ad29-a0759f1dea35" (UID: "7c9372f1-08ac-4d58-ad29-a0759f1dea35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:51:55.382497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.382480    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf" pod "7c9372f1-08ac-4d58-ad29-a0759f1dea35" (UID: "7c9372f1-08ac-4d58-ad29-a0759f1dea35"). InnerVolumeSpecName "kube-api-access-skbtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:55.481867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.481828    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbtf\" (UniqueName: \"kubernetes.io/projected/7c9372f1-08ac-4d58-ad29-a0759f1dea35-kube-api-access-skbtf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:55.481867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.481862    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:55.481867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:55.481872    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c9372f1-08ac-4d58-ad29-a0759f1dea35-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:51:56.070665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:56.070592    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"
Nov 03 00:51:56.071039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:56.070584    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb" event={"ID":"7c9372f1-08ac-4d58-ad29-a0759f1dea35","Type":"ContainerDied","Data":"c05221a1e4625309badbd3e577bb7b564b53048bcd8333f0946552fad3c5bb8a"}
Nov 03 00:51:56.071039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:56.070723    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05221a1e4625309badbd3e577bb7b564b53048bcd8333f0946552fad3c5bb8a"
Nov 03 00:51:56.663855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:56.663818    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:58.341571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:58.341529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:59.619548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:59.619515    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:59.743811 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:59.743770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:51:59.848873 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:51:59.848841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:00.113395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:00.113359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:00.213865 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:00.213833    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:00.381015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:00.380922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:01.049939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:01.049897    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:01.148196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:01.148154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:01.794218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:01.794183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:02.273957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:02.273917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:03.354388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:03.354356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:03.461464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:03.460975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:03.907519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:03.907478    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:04.238087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:04.237948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:04.983931 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:04.983900    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:05.083087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:05.083044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:05.186908 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:05.186864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:05.304084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:05.303991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:05.574849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:05.574759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:06.180318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:06.180276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:06.280024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:06.279981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:07.722723 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:07.722680    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:08.525989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:08.525958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:09.253128 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:09.253092    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:09.368889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:09.368854    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:09.847768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:09.847735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:09.948223 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:09.948184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:10.472753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:10.472718    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:11.005649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:11.005612    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:12.040932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:12.040902    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"]
Nov 03 00:52:12.045067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:12.045040    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ccfbee-wbx9j"]
Nov 03 00:52:12.200825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:12.200780    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:12.465724 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:12.465679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:13.044076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:13.044035    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:13.190917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:13.190879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:13.810359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:13.810320    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be9ed17-fa43-41f9-a174-f709b34092ba" path="/var/lib/kubelet/pods/2be9ed17-fa43-41f9-a174-f709b34092ba/volumes"
Nov 03 00:52:14.037627 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:14.037594    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:15.798806 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:15.798753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:15.920852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:15.920815    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:16.284230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:16.284195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:16.834463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:16.834398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:17.469701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:17.469662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:17.604940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:17.604910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:19.425798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:19.425761    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:20.483780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:20.483743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:20.589952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:20.589915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:21.256271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:21.256230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:22.033019 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.032981    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"]
Nov 03 00:52:22.046210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.046178    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"]
Nov 03 00:52:22.050348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.050322    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5343b68-pvtn6"]
Nov 03 00:52:22.054062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.054039    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5fce51-2p54p"]
Nov 03 00:52:22.144604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.144569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:22.252900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.252865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:22.316195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.316115    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"]
Nov 03 00:52:22.316892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.316873    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7c9372f1-08ac-4d58-ad29-a0759f1dea35" containerName="revoke-keys"
Nov 03 00:52:22.316892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.316891    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9372f1-08ac-4d58-ad29-a0759f1dea35" containerName="revoke-keys"
Nov 03 00:52:22.317042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.316994    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9372f1-08ac-4d58-ad29-a0759f1dea35" containerName="revoke-keys"
Nov 03 00:52:22.321274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.321252    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.336396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.336364    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-84f605-dockercfg-hwpvp\""
Nov 03 00:52:22.339824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.339800    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"]
Nov 03 00:52:22.371684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.371635    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.371895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.371743    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7lz\" (UniqueName: \"kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz\") pod \"payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.417423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.417387    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-84f605\""
Nov 03 00:52:22.520464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.489445    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.520464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.489552    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7lz\" (UniqueName: \"kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz\") pod \"payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.527457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.522092    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.567255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.567179    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7lz\" (UniqueName: \"kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz\") pod \"payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.642106 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.641714    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:22.983492 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:52:22.983456    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb62e20_9899_476b_a1a4_0db41756bbd1.slice/crio-4475fe069bc4dd5118927b2bd2f0fc2c3cb5fc62f128669d5fb6ef6d216e683c WatchSource:0}: Error finding container 4475fe069bc4dd5118927b2bd2f0fc2c3cb5fc62f128669d5fb6ef6d216e683c: Status 404 returned error can't find the container with id 4475fe069bc4dd5118927b2bd2f0fc2c3cb5fc62f128669d5fb6ef6d216e683c
Nov 03 00:52:22.985312 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:22.985227    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"]
Nov 03 00:52:23.041746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.041713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:23.619664 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.619627    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" event={"ID":"6fb62e20-9899-476b-a1a4-0db41756bbd1","Type":"ContainerStarted","Data":"a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c"}
Nov 03 00:52:23.619854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.619683    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" event={"ID":"6fb62e20-9899-476b-a1a4-0db41756bbd1","Type":"ContainerStarted","Data":"4475fe069bc4dd5118927b2bd2f0fc2c3cb5fc62f128669d5fb6ef6d216e683c"}
Nov 03 00:52:23.619854 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.619706    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:23.668834 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.668765    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" podStartSLOduration=1.668746683 podStartE2EDuration="1.668746683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:23.661521454 +0000 UTC m=+2825.983058279" watchObservedRunningTime="2026-11-03 00:52:23.668746683 +0000 UTC m=+2825.990283484"
Nov 03 00:52:23.823893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.823848    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09887ad0-66c2-48b5-9b75-34b6a4f4c313" path="/var/lib/kubelet/pods/09887ad0-66c2-48b5-9b75-34b6a4f4c313/volumes"
Nov 03 00:52:23.826299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:23.826256    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc567f22-d0d7-421f-8a05-5941f128ac44" path="/var/lib/kubelet/pods/fc567f22-d0d7-421f-8a05-5941f128ac44/volumes"
Nov 03 00:52:24.726448 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:52:24.726402    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:52:24.840362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:24.840327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:27.138043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.137997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:27.393957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.393855    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:27.526955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.526912    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:27.566943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.566595    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:52:27.566943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.566669    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:27.612458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.611750    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Nov 03 00:52:27.734973 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.717547    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:27.839184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.839142    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:27.918789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:27.918736    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:52:29.261236 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:29.261200    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:31.043373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:31.043334    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:32.831289 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:32.831254    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:33.679663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:33.679564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:33.778565 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:33.778425    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:34.068515 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:34.067492    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"]
Nov 03 00:52:34.091462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:34.081308    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f454281d-dsqtp"]
Nov 03 00:52:34.630548 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:34.630516    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 00:52:34.838659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:34.838571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:34.935847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:34.935814    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:35.076505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.072883    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"]
Nov 03 00:52:35.128471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.120283    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e69121a-jpw8s"]
Nov 03 00:52:35.199531 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.198899    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"]
Nov 03 00:52:35.220374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.220337    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.221268 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.221231    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"]
Nov 03 00:52:35.236841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.236812    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"kube-root-ca.crt\""
Nov 03 00:52:35.237016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.236903    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d4d8c2\""
Nov 03 00:52:35.237016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.236946    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"openshift-service-ca.crt\""
Nov 03 00:52:35.237016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.236999    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"routing-overlay\""
Nov 03 00:52:35.237175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.237037    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"agc-cred-83cf99d2ecc4\""
Nov 03 00:52:35.237175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.237083    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"payload-processing-external-model-e2e--b53073b3-dockercfg-t7plw\""
Nov 03 00:52:35.278004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.277962    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.278228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.278021    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.278228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.278121    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hp6w\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.278228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.278196    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.391950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.391896    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.405570 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.405529    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.413568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.413529    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hp6w\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.413881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.413862    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.441817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.393127    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.442420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.415276    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.450326 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.450230    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.493846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.493806    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hp6w\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.542401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.542369    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:35.809167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.809082    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1969f873-4481-4720-b8a8-f205af215e88" path="/var/lib/kubelet/pods/1969f873-4481-4720-b8a8-f205af215e88/volumes"
Nov 03 00:52:35.809601 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.809586    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b723a2-13ef-43ce-a3bf-b1e502091dd8" path="/var/lib/kubelet/pods/30b723a2-13ef-43ce-a3bf-b1e502091dd8/volumes"
Nov 03 00:52:35.869381 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:52:35.869345    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edb93de_b31b_4520_a01a_dc339dec498b.slice/crio-cacbb636a316beff1772320e7b0bdf9215932fd49a58769954fc8e66b5479fd0 WatchSource:0}: Error finding container cacbb636a316beff1772320e7b0bdf9215932fd49a58769954fc8e66b5479fd0: Status 404 returned error can't find the container with id cacbb636a316beff1772320e7b0bdf9215932fd49a58769954fc8e66b5479fd0
Nov 03 00:52:35.873363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:35.873334    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"]
Nov 03 00:52:36.525974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:36.525938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:36.888054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:36.887957    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr" event={"ID":"1edb93de-b31b-4520-a01a-dc339dec498b","Type":"ContainerStarted","Data":"656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd"}
Nov 03 00:52:36.888054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:36.888020    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr" event={"ID":"1edb93de-b31b-4520-a01a-dc339dec498b","Type":"ContainerStarted","Data":"cacbb636a316beff1772320e7b0bdf9215932fd49a58769954fc8e66b5479fd0"}
Nov 03 00:52:36.888054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:36.888048    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:36.927825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:36.927771    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr" podStartSLOduration=1.927757365 podStartE2EDuration="1.927757365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:36.92704391 +0000 UTC m=+2839.248580711" watchObservedRunningTime="2026-11-03 00:52:36.927757365 +0000 UTC m=+2839.249294163"
Nov 03 00:52:37.041287 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:37.041203    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"]
Nov 03 00:52:37.075552 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:37.074854    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e6dc5ee-9cwbw"]
Nov 03 00:52:37.809977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:37.809939    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f4bf15-547c-495f-95e8-1c64b6f1dadb" path="/var/lib/kubelet/pods/d9f4bf15-547c-495f-95e8-1c64b6f1dadb/volumes"
Nov 03 00:52:38.255911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:38.255868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:39.161649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.161612    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"]
Nov 03 00:52:39.167264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.167237    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.194555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.194520    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:52:39.194952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.194931    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wclrs\""
Nov 03 00:52:39.195166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.195151    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:52:39.195343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.195329    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:52:39.196855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.196824    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"]
Nov 03 00:52:39.280466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.280395    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.280645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.280564    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.280645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.280594    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.280645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.280614    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.280805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.280778    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.281059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.281027    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9rq\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.281109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.281094    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382256    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382458    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382513    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382547    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382818    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9rq\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.382917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.382873    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.383420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.383388    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.385091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.385064    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.385266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.385244    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.385507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.385484    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.385596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.385564    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.405527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.403685    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.405527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.403755    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9rq\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kzsj7\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.486722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.486639    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.812491 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:52:39.812421    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8578b459_8489_48d7_a768_85e55cfaef3d.slice/crio-ff43abfbaf8299f482f121138c703d9a19e417669e338086333de47080e3e42b WatchSource:0}: Error finding container ff43abfbaf8299f482f121138c703d9a19e417669e338086333de47080e3e42b: Status 404 returned error can't find the container with id ff43abfbaf8299f482f121138c703d9a19e417669e338086333de47080e3e42b
Nov 03 00:52:39.814705 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.814650    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"]
Nov 03 00:52:39.815888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.815854    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:39.815996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.815911    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:52:39.915907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.915868    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" event={"ID":"8578b459-8489-48d7-a768-85e55cfaef3d","Type":"ContainerStarted","Data":"6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997"}
Nov 03 00:52:39.916078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.915928    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" event={"ID":"8578b459-8489-48d7-a768-85e55cfaef3d","Type":"ContainerStarted","Data":"ff43abfbaf8299f482f121138c703d9a19e417669e338086333de47080e3e42b"}
Nov 03 00:52:39.916078 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.915955    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:39.952772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.952727    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:39.958699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:39.958642    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" podStartSLOduration=0.958624368 podStartE2EDuration="958.624368ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:39.957025182 +0000 UTC m=+2842.278561980" watchObservedRunningTime="2026-11-03 00:52:39.958624368 +0000 UTC m=+2842.280161168"
Nov 03 00:52:40.926782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:40.926749    4550 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kzsj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:52:40.927274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:40.926806    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:52:41.626180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:41.626141    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:43.317232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:43.317189    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:43.925717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:43.925683    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 00:52:45.024010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:45.023977    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:46.628607 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:46.628571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:47.901139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:47.901108    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:52:48.036065 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:48.036031    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"]
Nov 03 00:52:48.040466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:48.040412    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e9793a6-t74nm"]
Nov 03 00:52:48.336755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:48.336663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:49.031389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:49.031353    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"]
Nov 03 00:52:49.044079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:49.044046    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1d8715f-t622f"]
Nov 03 00:52:49.809081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:49.809049    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8a47ab-6458-44bd-a402-782199c96f10" path="/var/lib/kubelet/pods/8b8a47ab-6458-44bd-a402-782199c96f10/volumes"
Nov 03 00:52:49.809564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:49.809549    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb699652-7ada-46e0-9baa-a3f9fe9c957e" path="/var/lib/kubelet/pods/eb699652-7ada-46e0-9baa-a3f9fe9c957e/volumes"
Nov 03 00:52:49.962134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:49.962103    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:51.651177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:51.651136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:53.330989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:53.330956    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:54.724841 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:52:54.724805    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:52:55.012983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:55.012882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:56.619700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:56.619661    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:52:58.325260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:52:58.325225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:00.005374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:00.005335    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:01.751111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:01.751080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:03.585067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:03.585031    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:05.315095 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:05.315051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:05.412217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:05.412179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:05.518455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:05.518281    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:07.172808 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:07.172773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:08.922590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:08.922561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:10.526402 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:10.526356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:12.115994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:12.115958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:13.154313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:13.154274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:13.816768 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:13.816726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:15.431591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:15.431558    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:17.105752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:17.105714    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:18.705990 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:18.705953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:19.924690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:19.924654    4550 scope.go:122] "RemoveContainer" containerID="04fadcc566ad963ef60a1872bdf8db3a1269763641b54d084b8850861acf8759"
Nov 03 00:53:19.934974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:19.934949    4550 scope.go:122] "RemoveContainer" containerID="5b7e7a6a251c1ff52fb04a723cb451f908c0bc7a22c6d9bffb23ecc600b6aa8c"
Nov 03 00:53:19.944802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:19.944777    4550 scope.go:122] "RemoveContainer" containerID="ee6531e2da03b5f07ab30271838c37c3bf4cb0b6b2952d0d4ff0602641086ec6"
Nov 03 00:53:20.421417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:20.421371    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:22.017161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:22.017130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:23.700281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:23.700245    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:24.083244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:24.083161    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"]
Nov 03 00:53:24.083686 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:24.083659    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" containerID="cri-o://16904ce90ac94dc33383850e3a8eb1b0dd588595416668f16915bc6d1351f192" gracePeriod=30
Nov 03 00:53:24.725721 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:53:24.725679    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:53:25.617251 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:25.617214    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:27.413880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:27.413835    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:53:27.414376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:27.413921    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:53:27.465896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:27.465853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:29.192845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.192809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:29.369528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.368759    4550 generic.go:356] "Generic (PLEG): container finished" podID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerID="16904ce90ac94dc33383850e3a8eb1b0dd588595416668f16915bc6d1351f192" exitCode=0
Nov 03 00:53:29.369528 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.368833    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" event={"ID":"0bfdf551-5504-4e4b-8a40-954334350eb0","Type":"ContainerDied","Data":"16904ce90ac94dc33383850e3a8eb1b0dd588595416668f16915bc6d1351f192"}
Nov 03 00:53:29.438785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.438755    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:53:29.540325 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.540288    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.540549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.540350    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.540549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.540456    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.540683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.540600    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.540683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.540660    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:29.540817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.540793    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:29.541043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.541021    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.541404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.541363    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:29.541404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.541269    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.541589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.541486    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.541589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.541550    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.541719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.541591    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy\") pod \"0bfdf551-5504-4e4b-8a40-954334350eb0\" (UID: \"0bfdf551-5504-4e4b-8a40-954334350eb0\") "
Nov 03 00:53:29.542655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.542077    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:53:29.542655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.542260    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfdf551-5504-4e4b-8a40-954334350eb0-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.542655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.542279    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.542655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.542301    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.542655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.542314    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.542655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.542344    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:29.543101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.543078    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:53:29.543374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.543333    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "kube-api-access-n6g9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:53:29.544761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.544717    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:53:29.545965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.545940    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo" pod "0bfdf551-5504-4e4b-8a40-954334350eb0" (UID: "0bfdf551-5504-4e4b-8a40-954334350eb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:53:29.644275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.644190    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6g9b\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-kube-api-access-n6g9b\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.644275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.644222    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.644275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.644234    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.644275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.644243    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:29.644275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:29.644252    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfdf551-5504-4e4b-8a40-954334350eb0-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:53:30.379765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:30.379731    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"
Nov 03 00:53:30.379765 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:30.379748    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v" event={"ID":"0bfdf551-5504-4e4b-8a40-954334350eb0","Type":"ContainerDied","Data":"9655dc6db1717e9ec4152fb1c461296fb0b924e6c8d9ddfb74917b4298139f87"}
Nov 03 00:53:30.380306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:30.379830    4550 scope.go:122] "RemoveContainer" containerID="16904ce90ac94dc33383850e3a8eb1b0dd588595416668f16915bc6d1351f192"
Nov 03 00:53:30.448670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:30.448632    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"]
Nov 03 00:53:30.461773 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:30.461738    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-rp49v"]
Nov 03 00:53:30.932770 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:30.932738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:31.818978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:31.818944    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" path="/var/lib/kubelet/pods/0bfdf551-5504-4e4b-8a40-954334350eb0/volumes"
Nov 03 00:53:32.731760 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:32.731723    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:34.361374 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:34.361338    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:35.976127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:35.976095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:37.640638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:37.640604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:39.347771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:39.347728    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:41.206562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:41.206516    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:42.881488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:42.881450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:44.568350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:44.568315    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:46.312358 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:46.312323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:47.997014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:47.996975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:49.643572 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:49.643523    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:51.234147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:51.234112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:52.942776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:52.942741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:54.716139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:54.716090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:54.724728 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:53:54.724694    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:53:55.040875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:55.040771    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"]
Nov 03 00:53:55.053184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:55.053149    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e0da59f-lb9tp"]
Nov 03 00:53:55.810002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:55.809965    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41c1c337-d01f-4cec-a974-3a74ab12bce5" path="/var/lib/kubelet/pods/41c1c337-d01f-4cec-a974-3a74ab12bce5/volumes"
Nov 03 00:53:56.412974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:56.412939    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:57.671278 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:57.671244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:58.130446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:58.130347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:58.231148 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:58.231110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:58.599939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:58.599902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:58.704581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:58.704541    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:53:59.930881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:53:59.930781    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:01.586776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:01.586739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:02.458256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.458214    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"]
Nov 03 00:54:02.459170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.459147    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy"
Nov 03 00:54:02.459170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.459171    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy"
Nov 03 00:54:02.459380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.459329    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfdf551-5504-4e4b-8a40-954334350eb0" containerName="istio-proxy"
Nov 03 00:54:02.463293 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.463267    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.481112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.481078    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"]
Nov 03 00:54:02.489366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.489329    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:54:02.489568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.489384    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:54:02.610957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.610919    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.611475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.611154    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngmg\" (UniqueName: \"kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.611475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.611327    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.714233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.714139    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.714498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.714250    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.714498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.714385    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngmg\" (UniqueName: \"kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.715001 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.714979    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.717423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.717393    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.734550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.734518    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngmg\" (UniqueName: \"kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg\") pod \"maas-api-revoke-sub-b0050a8e-wjwfv\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.780232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.780185    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:02.813474 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:02.813427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:03.043152 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:03.043064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:03.094800 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:54:03.094765    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd273ce19_d14e_452e_a2d2_831ce855f063.slice/crio-bf412670ab22fe6e1acf4365cfcdcd3b4a03a4815acd201dc28c6ae027174de3 WatchSource:0}: Error finding container bf412670ab22fe6e1acf4365cfcdcd3b4a03a4815acd201dc28c6ae027174de3: Status 404 returned error can't find the container with id bf412670ab22fe6e1acf4365cfcdcd3b4a03a4815acd201dc28c6ae027174de3
Nov 03 00:54:03.096596 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:03.096569    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"]
Nov 03 00:54:03.550240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:03.550206    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:03.729754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:03.729714    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv" event={"ID":"d273ce19-d14e-452e-a2d2-831ce855f063","Type":"ContainerStarted","Data":"528431011aeaa00b806ff488bdfd3e9b00c65c60be3f5bea4c9f9afc12bec9ef"}
Nov 03 00:54:03.730244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:03.729818    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv" event={"ID":"d273ce19-d14e-452e-a2d2-831ce855f063","Type":"ContainerStarted","Data":"bf412670ab22fe6e1acf4365cfcdcd3b4a03a4815acd201dc28c6ae027174de3"}
Nov 03 00:54:03.768646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:03.768589    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv" podStartSLOduration=1.768573607 podStartE2EDuration="1.768573607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:03.76833517 +0000 UTC m=+2926.089871971" watchObservedRunningTime="2026-11-03 00:54:03.768573607 +0000 UTC m=+2926.090110406"
Nov 03 00:54:04.457330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:04.457293    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:04.559125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:04.559088    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:04.968898 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:04.968831    4550 generic.go:356] "Generic (PLEG): container finished" podID="d273ce19-d14e-452e-a2d2-831ce855f063" containerID="528431011aeaa00b806ff488bdfd3e9b00c65c60be3f5bea4c9f9afc12bec9ef" exitCode=0
Nov 03 00:54:04.969309 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:04.969180    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv" event={"ID":"d273ce19-d14e-452e-a2d2-831ce855f063","Type":"ContainerDied","Data":"528431011aeaa00b806ff488bdfd3e9b00c65c60be3f5bea4c9f9afc12bec9ef"}
Nov 03 00:54:05.375850 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:05.375760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:06.151603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.151575    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:06.288494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.288392    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg\" (UniqueName: \"kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg\") pod \"d273ce19-d14e-452e-a2d2-831ce855f063\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") "
Nov 03 00:54:06.288654 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.288595    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca\") pod \"d273ce19-d14e-452e-a2d2-831ce855f063\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") "
Nov 03 00:54:06.288805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.288777    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca\") pod \"d273ce19-d14e-452e-a2d2-831ce855f063\" (UID: \"d273ce19-d14e-452e-a2d2-831ce855f063\") "
Nov 03 00:54:06.289039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.289006    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca" pod "d273ce19-d14e-452e-a2d2-831ce855f063" (UID: "d273ce19-d14e-452e-a2d2-831ce855f063"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:06.289491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.289465    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:06.290703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.290675    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg" pod "d273ce19-d14e-452e-a2d2-831ce855f063" (UID: "d273ce19-d14e-452e-a2d2-831ce855f063"). InnerVolumeSpecName "kube-api-access-bngmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:06.290814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.290740    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca" pod "d273ce19-d14e-452e-a2d2-831ce855f063" (UID: "d273ce19-d14e-452e-a2d2-831ce855f063"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:54:06.390591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.390559    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d273ce19-d14e-452e-a2d2-831ce855f063-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:06.390591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.390588    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bngmg\" (UniqueName: \"kubernetes.io/projected/d273ce19-d14e-452e-a2d2-831ce855f063-kube-api-access-bngmg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:06.985640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.985605    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"
Nov 03 00:54:06.985859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.985607    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv" event={"ID":"d273ce19-d14e-452e-a2d2-831ce855f063","Type":"ContainerDied","Data":"bf412670ab22fe6e1acf4365cfcdcd3b4a03a4815acd201dc28c6ae027174de3"}
Nov 03 00:54:06.985859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:06.985734    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf412670ab22fe6e1acf4365cfcdcd3b4a03a4815acd201dc28c6ae027174de3"
Nov 03 00:54:07.135196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:07.135160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:08.029867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:08.029825    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"]
Nov 03 00:54:08.043835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:08.043802    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d44d572a-4lccw"]
Nov 03 00:54:08.865441 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:08.865392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:09.810250 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:09.810212    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dd7b05-7d9e-4853-90b6-7415e3496e31" path="/var/lib/kubelet/pods/65dd7b05-7d9e-4853-90b6-7415e3496e31/volumes"
Nov 03 00:54:10.569527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:10.569489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:12.122115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:12.122075    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:13.284818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:13.284779    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:13.738369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:13.738296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:13.746341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:13.746298    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"]
Nov 03 00:54:13.843367 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:13.843317    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:13.947092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:13.947061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:14.369498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.369459    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"]
Nov 03 00:54:14.370233 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.370216    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d273ce19-d14e-452e-a2d2-831ce855f063" containerName="revoke-keys"
Nov 03 00:54:14.370297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.370235    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d273ce19-d14e-452e-a2d2-831ce855f063" containerName="revoke-keys"
Nov 03 00:54:14.370359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.370350    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d273ce19-d14e-452e-a2d2-831ce855f063" containerName="revoke-keys"
Nov 03 00:54:14.374456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.374418    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.390598 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.390565    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:54:14.391210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.391192    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:54:14.394982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.394953    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"]
Nov 03 00:54:14.435907 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.435850    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9c95\" (UniqueName: \"kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.436183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.436155    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.436281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.436209    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.537859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.537805    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.538086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.537878    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.538086 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.537977    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9c95\" (UniqueName: \"kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.538620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.538593    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.540296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.540272    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.558603 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.558565    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9c95\" (UniqueName: \"kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95\") pod \"maas-api-revoke-sub-924984da-l57v2\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.692118 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:14.692079    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:14.999828 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:54:14.999787    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e47dee8_72a2_4214_afa4_94dba004e66f.slice/crio-d6b0812d5e06d8a79700aa8bd207925b39025d0a1b8823554e0326ebe33aef72 WatchSource:0}: Error finding container d6b0812d5e06d8a79700aa8bd207925b39025d0a1b8823554e0326ebe33aef72: Status 404 returned error can't find the container with id d6b0812d5e06d8a79700aa8bd207925b39025d0a1b8823554e0326ebe33aef72
Nov 03 00:54:15.002960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:15.002927    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"]
Nov 03 00:54:15.030665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:15.030634    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"]
Nov 03 00:54:15.037521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:15.037482    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b271f0c-vrwd6"]
Nov 03 00:54:15.053869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:15.053828    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2" event={"ID":"7e47dee8-72a2-4214-afa4-94dba004e66f","Type":"ContainerStarted","Data":"d6b0812d5e06d8a79700aa8bd207925b39025d0a1b8823554e0326ebe33aef72"}
Nov 03 00:54:15.666938 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:15.666902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:15.813101 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:15.813071    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f51d6e-439d-4a08-be58-32a14c67cac2" path="/var/lib/kubelet/pods/77f51d6e-439d-4a08-be58-32a14c67cac2/volumes"
Nov 03 00:54:16.065191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:16.065154    4550 generic.go:356] "Generic (PLEG): container finished" podID="7e47dee8-72a2-4214-afa4-94dba004e66f" containerID="872daa101ecc38988c5f3659fb63ba1a30128e7ea34de78bc96a90c58d30a106" exitCode=0
Nov 03 00:54:16.065422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:16.065246    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2" event={"ID":"7e47dee8-72a2-4214-afa4-94dba004e66f","Type":"ContainerDied","Data":"872daa101ecc38988c5f3659fb63ba1a30128e7ea34de78bc96a90c58d30a106"}
Nov 03 00:54:17.265928 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.265897    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:17.392513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.392053    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca\") pod \"7e47dee8-72a2-4214-afa4-94dba004e66f\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") "
Nov 03 00:54:17.392513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.392406    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca\") pod \"7e47dee8-72a2-4214-afa4-94dba004e66f\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") "
Nov 03 00:54:17.392513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.392469    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95\" (UniqueName: \"kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95\") pod \"7e47dee8-72a2-4214-afa4-94dba004e66f\" (UID: \"7e47dee8-72a2-4214-afa4-94dba004e66f\") "
Nov 03 00:54:17.395208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.395176    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca" pod "7e47dee8-72a2-4214-afa4-94dba004e66f" (UID: "7e47dee8-72a2-4214-afa4-94dba004e66f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:17.398680 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.398321    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:17.402637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.402592    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95" pod "7e47dee8-72a2-4214-afa4-94dba004e66f" (UID: "7e47dee8-72a2-4214-afa4-94dba004e66f"). InnerVolumeSpecName "kube-api-access-n9c95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:17.404217 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.404175    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca" pod "7e47dee8-72a2-4214-afa4-94dba004e66f" (UID: "7e47dee8-72a2-4214-afa4-94dba004e66f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:54:17.499388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.499300    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:17.499388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.499343    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9c95\" (UniqueName: \"kubernetes.io/projected/7e47dee8-72a2-4214-afa4-94dba004e66f-kube-api-access-n9c95\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:17.499388 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:17.499357    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e47dee8-72a2-4214-afa4-94dba004e66f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:18.089785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:18.089699    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"
Nov 03 00:54:18.089955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:18.089693    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2" event={"ID":"7e47dee8-72a2-4214-afa4-94dba004e66f","Type":"ContainerDied","Data":"d6b0812d5e06d8a79700aa8bd207925b39025d0a1b8823554e0326ebe33aef72"}
Nov 03 00:54:18.089955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:18.089836    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b0812d5e06d8a79700aa8bd207925b39025d0a1b8823554e0326ebe33aef72"
Nov 03 00:54:18.747201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:18.747152    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr" podUID="1edb93de-b31b-4520-a01a-dc339dec498b" containerName="payload-processing" containerID="cri-o://656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd" gracePeriod=25
Nov 03 00:54:19.021642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.021613    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:19.056488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.056425    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:54:19.137331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.137266    4550 generic.go:356] "Generic (PLEG): container finished" podID="1edb93de-b31b-4520-a01a-dc339dec498b" containerID="656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd" exitCode=0
Nov 03 00:54:19.139807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.139770    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr" event={"ID":"1edb93de-b31b-4520-a01a-dc339dec498b","Type":"ContainerDied","Data":"656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd"}
Nov 03 00:54:19.139979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.139834    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr" event={"ID":"1edb93de-b31b-4520-a01a-dc339dec498b","Type":"ContainerDied","Data":"cacbb636a316beff1772320e7b0bdf9215932fd49a58769954fc8e66b5479fd0"}
Nov 03 00:54:19.139979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.139862    4550 scope.go:122] "RemoveContainer" containerID="656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd"
Nov 03 00:54:19.140134 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.140027    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"
Nov 03 00:54:19.158124 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.158022    4550 scope.go:122] "RemoveContainer" containerID="656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd"
Nov 03 00:54:19.158711 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:54:19.158673    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd\": container with ID starting with 656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd not found: ID does not exist" containerID="656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd"
Nov 03 00:54:19.158866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.158725    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd"} err="failed to get container status \"656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd\": rpc error: code = NotFound desc = could not find container \"656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd\": container with ID starting with 656d2d9fe3df9e2cd79e2d712a8fb4f9372c3996881b738b41a8425e789e45fd not found: ID does not exist"
Nov 03 00:54:19.232182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.232138    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w\") pod \"1edb93de-b31b-4520-a01a-dc339dec498b\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") "
Nov 03 00:54:19.232345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.232203    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume\") pod \"1edb93de-b31b-4520-a01a-dc339dec498b\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") "
Nov 03 00:54:19.232468 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.232377    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials\") pod \"1edb93de-b31b-4520-a01a-dc339dec498b\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") "
Nov 03 00:54:19.232615 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.232598    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume\") pod \"1edb93de-b31b-4520-a01a-dc339dec498b\" (UID: \"1edb93de-b31b-4520-a01a-dc339dec498b\") "
Nov 03 00:54:19.232697 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.232659    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume" pod "1edb93de-b31b-4520-a01a-dc339dec498b" (UID: "1edb93de-b31b-4520-a01a-dc339dec498b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:19.232974 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.232953    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume" pod "1edb93de-b31b-4520-a01a-dc339dec498b" (UID: "1edb93de-b31b-4520-a01a-dc339dec498b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:19.233190 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.233173    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:19.233244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.233196    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1edb93de-b31b-4520-a01a-dc339dec498b-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:19.234631 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.234605    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials" pod "1edb93de-b31b-4520-a01a-dc339dec498b" (UID: "1edb93de-b31b-4520-a01a-dc339dec498b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:19.234720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.234648    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w" pod "1edb93de-b31b-4520-a01a-dc339dec498b" (UID: "1edb93de-b31b-4520-a01a-dc339dec498b"). InnerVolumeSpecName "kube-api-access-8hp6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:19.335125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.335025    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hp6w\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-kube-api-access-8hp6w\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:19.335125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.335063    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1edb93de-b31b-4520-a01a-dc339dec498b-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:19.493313 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.493275    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"]
Nov 03 00:54:19.503555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.503523    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-64bdvlbbr"]
Nov 03 00:54:19.810857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.810820    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edb93de-b31b-4520-a01a-dc339dec498b" path="/var/lib/kubelet/pods/1edb93de-b31b-4520-a01a-dc339dec498b/volumes"
Nov 03 00:54:19.991373 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:19.991237    4550 scope.go:122] "RemoveContainer" containerID="4e730e2872c89e60374430949907145ead3be9e6726fb532a01de7c380bc4d66"
Nov 03 00:54:20.004196 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:20.004137    4550 scope.go:122] "RemoveContainer" containerID="6895376fcb2d024cedcbdb527a5b3d1f7879fc4ad3f10a7a51103944d83fc791"
Nov 03 00:54:20.016219 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:20.016194    4550 scope.go:122] "RemoveContainer" containerID="0b2f40a533782c28d5e595337aae2a74d21669f06e1fcae7d8ad4bd0db4e9534"
Nov 03 00:54:20.026205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:20.026176    4550 scope.go:122] "RemoveContainer" containerID="fdf8141e2a057bbc5822b2201fa44d98d43e831f8b0d5e2e47a995f165d3bb11"
Nov 03 00:54:20.036666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:20.036643    4550 scope.go:122] "RemoveContainer" containerID="da37f800fb23b980c2e87a5d6ebe48f45834a929e5d4a59e0331cf76663a7d0f"
Nov 03 00:54:20.772146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:20.772098    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:22.423762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:22.423723    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:24.017530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:24.017490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:24.727931 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:54:24.727880    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:54:25.604473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:25.603697    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:25.694866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:25.694778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:25.795676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:25.795630    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:26.191456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:26.191196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:27.415782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:27.415740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:29.043186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:29.043141    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:30.755702 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:30.755669    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:32.030980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:32.030943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:32.140454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:32.138713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:32.276822 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:32.276788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:32.502530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:32.502492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:32.619425 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:32.619377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:32.804701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:32.804604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:33.414120 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:33.414081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:33.514919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:33.514883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:33.865513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:33.864643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:34.457163 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:34.457127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:35.321255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:35.321221    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:36.352467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:36.352394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:36.646970 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:36.646884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:38.455241 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:38.455200    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:40.053561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:40.053524    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:40.675202 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:40.675146    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:40.768594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:40.768551    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:41.306987 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:41.306954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:41.475011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:41.474975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:41.978023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:41.977987    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:43.670090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:43.670057    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:45.330870 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:45.330830    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:45.424786 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:45.424739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:47.006214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:47.006176    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:47.404481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:47.404448    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:47.504115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:47.504080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:47.881227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:47.881129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:48.359024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.358988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:48.507674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.507635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:48.746459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.745212    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"]
Nov 03 00:54:48.776417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.776367    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7e47dee8-72a2-4214-afa4-94dba004e66f" containerName="revoke-keys"
Nov 03 00:54:48.776417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.776405    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e47dee8-72a2-4214-afa4-94dba004e66f" containerName="revoke-keys"
Nov 03 00:54:48.776417 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.776449    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1edb93de-b31b-4520-a01a-dc339dec498b" containerName="payload-processing"
Nov 03 00:54:48.776756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.776458    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edb93de-b31b-4520-a01a-dc339dec498b" containerName="payload-processing"
Nov 03 00:54:48.776756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.776678    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edb93de-b31b-4520-a01a-dc339dec498b" containerName="payload-processing"
Nov 03 00:54:48.776756 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.776705    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e47dee8-72a2-4214-afa4-94dba004e66f" containerName="revoke-keys"
Nov 03 00:54:48.937126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.931006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:48.940852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.939180    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"]
Nov 03 00:54:48.940852 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.939645    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:48.959504 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.959474    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:54:48.980343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:48.980300    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:54:49.052408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.052108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:49.056696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.053289    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.057470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.057411    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.057817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.057754    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzqk\" (UniqueName: \"kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.145959 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.145918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:49.183568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.183518    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzqk\" (UniqueName: \"kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.184140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.184109    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.184523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.184491    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.185207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.185181    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.190698 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.190670    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.210253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.210212    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzqk\" (UniqueName: \"kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk\") pod \"maas-api-revoke-sub-b2ff8ac3-7m9tj\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.313720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.313629    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:49.634171 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:54:49.634134    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e3865f_38e4_4eb5_af83_71171c64cda0.slice/crio-4674e993671b88d8abee3b86e4c181bd01ebac8a2c2fba9f7457663936cdd08f WatchSource:0}: Error finding container 4674e993671b88d8abee3b86e4c181bd01ebac8a2c2fba9f7457663936cdd08f: Status 404 returned error can't find the container with id 4674e993671b88d8abee3b86e4c181bd01ebac8a2c2fba9f7457663936cdd08f
Nov 03 00:54:49.635538 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:49.635215    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"]
Nov 03 00:54:50.486772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:50.486722    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj" event={"ID":"47e3865f-38e4-4eb5-af83-71171c64cda0","Type":"ContainerStarted","Data":"a8cec05a77d3802a11eca82804b18175852c4161fbffc0c702c0017c77452f8f"}
Nov 03 00:54:50.486969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:50.486804    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj" event={"ID":"47e3865f-38e4-4eb5-af83-71171c64cda0","Type":"ContainerStarted","Data":"4674e993671b88d8abee3b86e4c181bd01ebac8a2c2fba9f7457663936cdd08f"}
Nov 03 00:54:50.527345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:50.527278    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj" podStartSLOduration=2.527258138 podStartE2EDuration="2.527258138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:50.526360944 +0000 UTC m=+2972.847897745" watchObservedRunningTime="2026-11-03 00:54:50.527258138 +0000 UTC m=+2972.848794942"
Nov 03 00:54:50.809945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:50.809911    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:51.496757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:51.496717    4550 generic.go:356] "Generic (PLEG): container finished" podID="47e3865f-38e4-4eb5-af83-71171c64cda0" containerID="a8cec05a77d3802a11eca82804b18175852c4161fbffc0c702c0017c77452f8f" exitCode=0
Nov 03 00:54:51.496965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:51.496862    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj" event={"ID":"47e3865f-38e4-4eb5-af83-71171c64cda0","Type":"ContainerDied","Data":"a8cec05a77d3802a11eca82804b18175852c4161fbffc0c702c0017c77452f8f"}
Nov 03 00:54:52.277913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.277856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:52.383782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.383742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:52.692960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.692928    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:52.722982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.722947    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:52.777456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.763515    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca\") pod \"47e3865f-38e4-4eb5-af83-71171c64cda0\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") "
Nov 03 00:54:52.777456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.764040    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca" pod "47e3865f-38e4-4eb5-af83-71171c64cda0" (UID: "47e3865f-38e4-4eb5-af83-71171c64cda0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:52.777456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.764690    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca\") pod \"47e3865f-38e4-4eb5-af83-71171c64cda0\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") "
Nov 03 00:54:52.777456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.764916    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk\" (UniqueName: \"kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk\") pod \"47e3865f-38e4-4eb5-af83-71171c64cda0\" (UID: \"47e3865f-38e4-4eb5-af83-71171c64cda0\") "
Nov 03 00:54:52.777456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.766967    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:52.823297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.819633    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca" pod "47e3865f-38e4-4eb5-af83-71171c64cda0" (UID: "47e3865f-38e4-4eb5-af83-71171c64cda0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:54:52.846492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.843728    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk" pod "47e3865f-38e4-4eb5-af83-71171c64cda0" (UID: "47e3865f-38e4-4eb5-af83-71171c64cda0"). InnerVolumeSpecName "kube-api-access-bwzqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:52.869558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.869523    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwzqk\" (UniqueName: \"kubernetes.io/projected/47e3865f-38e4-4eb5-af83-71171c64cda0-kube-api-access-bwzqk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:52.869558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:52.869555    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e3865f-38e4-4eb5-af83-71171c64cda0-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:54:53.033847 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.033812    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"]
Nov 03 00:54:53.037960 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.037933    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4442b5e0-pkdpk"]
Nov 03 00:54:53.516571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.516541    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"
Nov 03 00:54:53.517039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.516590    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj" event={"ID":"47e3865f-38e4-4eb5-af83-71171c64cda0","Type":"ContainerDied","Data":"4674e993671b88d8abee3b86e4c181bd01ebac8a2c2fba9f7457663936cdd08f"}
Nov 03 00:54:53.517039 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.516650    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4674e993671b88d8abee3b86e4c181bd01ebac8a2c2fba9f7457663936cdd08f"
Nov 03 00:54:53.814131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.814054    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad5aa74-dcd3-453d-b079-b5e1aefcb920" path="/var/lib/kubelet/pods/dad5aa74-dcd3-453d-b079-b5e1aefcb920/volumes"
Nov 03 00:54:53.929403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:53.929369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:54.038524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:54.038491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:54.634256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:54.634224    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:54.755630 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:54:54.755593    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:54:56.355512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:56.355473    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:58.123940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:58.123898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:59.801271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:59.801238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:54:59.969610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:54:59.969570    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:00.056307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.056210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:00.876491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.865519    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"]
Nov 03 00:55:00.876491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.866586    4550 cpu_manager.go:414] "containerMap: removing container" podUID="47e3865f-38e4-4eb5-af83-71171c64cda0" containerName="revoke-keys"
Nov 03 00:55:00.876491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.866606    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e3865f-38e4-4eb5-af83-71171c64cda0" containerName="revoke-keys"
Nov 03 00:55:00.876491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.866773    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e3865f-38e4-4eb5-af83-71171c64cda0" containerName="revoke-keys"
Nov 03 00:55:00.876491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.871264    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:00.891642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.891396    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:55:00.910454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.902759    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"]
Nov 03 00:55:00.910741 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.905474    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:55:00.920061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.920021    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khs8\" (UniqueName: \"kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:00.920699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.920663    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:00.920868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:00.920850    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.024161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.022845    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.024161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.022909    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.024161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.023059    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khs8\" (UniqueName: \"kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.081455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.061287    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.082995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.082908    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.124454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.123150    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khs8\" (UniqueName: \"kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8\") pod \"maas-api-revoke-sub-e3d748c2-zwhxd\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.281027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.280986    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:01.587709 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:55:01.587676    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8411618_78bf_4398_b7f4_3326495a8691.slice/crio-ddad119a69828513c1a4418e4b6c099cc2e93cd7739617014d978da3a97959f0 WatchSource:0}: Error finding container ddad119a69828513c1a4418e4b6c099cc2e93cd7739617014d978da3a97959f0: Status 404 returned error can't find the container with id ddad119a69828513c1a4418e4b6c099cc2e93cd7739617014d978da3a97959f0
Nov 03 00:55:01.589455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.589421    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"]
Nov 03 00:55:01.603855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.603828    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd" event={"ID":"d8411618-78bf-4398-b7f4-3326495a8691","Type":"ContainerStarted","Data":"ddad119a69828513c1a4418e4b6c099cc2e93cd7739617014d978da3a97959f0"}
Nov 03 00:55:01.925397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:01.925362    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:02.614167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:02.614130    4550 generic.go:356] "Generic (PLEG): container finished" podID="d8411618-78bf-4398-b7f4-3326495a8691" containerID="189598308b8c8f0f66f0178830647e8280625ef95443d14715d313b84f7ed010" exitCode=0
Nov 03 00:55:02.614376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:02.614216    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd" event={"ID":"d8411618-78bf-4398-b7f4-3326495a8691","Type":"ContainerDied","Data":"189598308b8c8f0f66f0178830647e8280625ef95443d14715d313b84f7ed010"}
Nov 03 00:55:03.621027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.620983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:03.854967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.854940    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:03.869290 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.869244    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8\" (UniqueName: \"kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8\") pod \"d8411618-78bf-4398-b7f4-3326495a8691\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") "
Nov 03 00:55:03.869533 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.869509    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca\") pod \"d8411618-78bf-4398-b7f4-3326495a8691\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") "
Nov 03 00:55:03.869701 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.869683    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca\") pod \"d8411618-78bf-4398-b7f4-3326495a8691\" (UID: \"d8411618-78bf-4398-b7f4-3326495a8691\") "
Nov 03 00:55:03.870407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.870365    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca" pod "d8411618-78bf-4398-b7f4-3326495a8691" (UID: "d8411618-78bf-4398-b7f4-3326495a8691"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:03.877549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.876672    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8" pod "d8411618-78bf-4398-b7f4-3326495a8691" (UID: "d8411618-78bf-4398-b7f4-3326495a8691"). InnerVolumeSpecName "kube-api-access-6khs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:03.877549 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.876736    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca" pod "d8411618-78bf-4398-b7f4-3326495a8691" (UID: "d8411618-78bf-4398-b7f4-3326495a8691"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:55:03.962892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.962687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:03.976481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.971675    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:55:03.976481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.971719    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8411618-78bf-4398-b7f4-3326495a8691-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:55:03.976481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:03.971735    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6khs8\" (UniqueName: \"kubernetes.io/projected/d8411618-78bf-4398-b7f4-3326495a8691-kube-api-access-6khs8\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:55:04.061767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:04.061726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:04.291457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:04.291072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:04.691049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:04.685764    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd" event={"ID":"d8411618-78bf-4398-b7f4-3326495a8691","Type":"ContainerDied","Data":"ddad119a69828513c1a4418e4b6c099cc2e93cd7739617014d978da3a97959f0"}
Nov 03 00:55:04.691049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:04.685847    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddad119a69828513c1a4418e4b6c099cc2e93cd7739617014d978da3a97959f0"
Nov 03 00:55:04.691049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:04.685941    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"
Nov 03 00:55:04.925354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:04.925321    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:05.811542 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:05.811504    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:07.456320 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:07.456274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:09.164738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:09.164702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:09.634574 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:09.634485    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:10.822740 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:10.822703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:12.219755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:12.208324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:12.299855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:12.299819    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:12.630408 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:12.630323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:12.951506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:12.907713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:13.178563 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:13.177000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:13.542363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:13.542328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:13.815495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:13.815357    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:14.103626 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:14.103524    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:14.208111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:14.208071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:14.483282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:14.480251    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:14.745582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:14.741079    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:15.001832 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:15.001741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:15.677455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:15.674898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:15.772175 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:15.772131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:17.381677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:17.381632    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:19.061571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:19.061535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:20.121285 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:20.121257    4550 scope.go:122] "RemoveContainer" containerID="0b0dadd40158588c693ea5bb2a8d8a1d267ba84dfa71a5f6ff9738b1d9568326"
Nov 03 00:55:20.133452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:20.133416    4550 scope.go:122] "RemoveContainer" containerID="815cc974e4f54c1f2d409478f89309336e306d8800342627e17a5ef58c4950e6"
Nov 03 00:55:20.143797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:20.143774    4550 scope.go:122] "RemoveContainer" containerID="54c0d65fe85fd982177d625954e7c2be11651c97bccb19075f2833026e6df30b"
Nov 03 00:55:20.505817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:20.505788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:20.607243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:20.607210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:20.826395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:20.826309    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:22.424275 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:22.424227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:24.069164 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:24.069130    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:24.724688 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:55:24.724655    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:55:25.687237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:25.687184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:26.309777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:26.309736    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:26.403187 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:26.403153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:26.778514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:26.778482    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:27.218735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.218699    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"]
Nov 03 00:55:27.219863 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.219835    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d8411618-78bf-4398-b7f4-3326495a8691" containerName="revoke-keys"
Nov 03 00:55:27.219986 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.219865    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8411618-78bf-4398-b7f4-3326495a8691" containerName="revoke-keys"
Nov 03 00:55:27.220108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.220093    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8411618-78bf-4398-b7f4-3326495a8691" containerName="revoke-keys"
Nov 03 00:55:27.226463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.226442    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.249048 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.249017    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"]
Nov 03 00:55:27.253239 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.253199    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-49d7e5-openshift-default-dockercfg-98pqq\""
Nov 03 00:55:27.276379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.276340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:27.346994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.346964    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347000    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347156    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347210 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347185    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347406    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347468    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347514    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347584    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.347638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.347616    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.375894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.375863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:27.449385 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449352    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449390    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449463    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449509    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449543    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449643    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449673    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449804    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449805    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.449845 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449829    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.450139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.449839    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.450205 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.450133    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.450254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.450204    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.450339 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.450317    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.451957 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.451930    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.452188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.452169    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.497562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.495070    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:27.497738 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.497594    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.528453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.521170    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7\") pod \"e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") " pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:27.552184 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:27.551859    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:28.130568 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:55:28.130531    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode203d55a_e393_443b_b9f3_ac898417f112.slice/crio-7382b5ed34da39c09f6a178158797edb04bdd5c77671274263cf46c36eb93dbd WatchSource:0}: Error finding container 7382b5ed34da39c09f6a178158797edb04bdd5c77671274263cf46c36eb93dbd: Status 404 returned error can't find the container with id 7382b5ed34da39c09f6a178158797edb04bdd5c77671274263cf46c36eb93dbd
Nov 03 00:55:28.132212 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:28.132187    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"]
Nov 03 00:55:28.134087 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:28.134052    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:28.134201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:28.134129    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:28.134201 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:28.134174    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:55:28.983227 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:28.983195    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" event={"ID":"e203d55a-e393-443b-b9f3-ac898417f112","Type":"ContainerStarted","Data":"2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"}
Nov 03 00:55:28.983472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:28.983255    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" event={"ID":"e203d55a-e393-443b-b9f3-ac898417f112","Type":"ContainerStarted","Data":"7382b5ed34da39c09f6a178158797edb04bdd5c77671274263cf46c36eb93dbd"}
Nov 03 00:55:29.055892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:29.055832    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podStartSLOduration=2.055815266 podStartE2EDuration="2.055815266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:29.039392271 +0000 UTC m=+3011.360929082" watchObservedRunningTime="2026-11-03 00:55:29.055815266 +0000 UTC m=+3011.377352069"
Nov 03 00:55:29.152330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:29.152300    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:29.552687 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:29.552596    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:29.560405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:29.560381    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:29.991849 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:29.991811    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:29.993036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:29.992999    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:55:30.943322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:30.943291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:31.435663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:31.435625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:31.546695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:31.546653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:32.501875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:32.501842    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:32.600161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:32.600110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:32.812488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:32.812352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:33.206025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:33.205991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:33.305785 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:33.305748    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:34.692206 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:34.692171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:36.514975 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:36.514938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:38.487646 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:38.487616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:40.230799 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:40.230764    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:41.942077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:41.942038    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:43.030875 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:43.030836    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"]
Nov 03 00:55:43.043835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:43.043797    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7a50f76-lf7fj"]
Nov 03 00:55:43.641716 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:43.641672    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:43.741390 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:43.741354    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:43.809691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:43.809657    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5246569a-89e7-4734-9b33-27b51f274531" path="/var/lib/kubelet/pods/5246569a-89e7-4734-9b33-27b51f274531/volumes"
Nov 03 00:55:43.845090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:43.844943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:45.100795 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:45.100763    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:45.494678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:45.494638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:46.439988 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:46.439926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:47.279946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:47.279903    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:48.955846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:48.955805    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:50.642665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:50.642629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:52.411183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:52.411148    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:54.038711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:54.038676    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:54.724481 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:55:54.724416    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:55:55.030090 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:55.030016    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"]
Nov 03 00:55:55.042800 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:55.042770    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14627cb-pmwfj"]
Nov 03 00:55:55.633678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:55.633637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:55.810892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:55.810859    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271a04b5-aff2-475f-a4be-2d4e7a34313f" path="/var/lib/kubelet/pods/271a04b5-aff2-475f-a4be-2d4e7a34313f/volumes"
Nov 03 00:55:57.405980 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:57.405949    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:55:59.043801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:55:59.043765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:01.023772 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:01.023739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:02.907396 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:02.907359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:04.536828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:04.536794    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:06.030711 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:06.030662    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"]
Nov 03 00:56:06.043228 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:06.043198    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-216209e3-xzqp9"]
Nov 03 00:56:06.261142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:06.261107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:07.811782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:07.811750    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e617fc-b30c-4df8-ba73-fde2083cbf72" path="/var/lib/kubelet/pods/60e617fc-b30c-4df8-ba73-fde2083cbf72/volumes"
Nov 03 00:56:08.032945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:08.032904    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:08.674577 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:08.674528    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:09.844397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:09.844365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:11.530520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:11.530477    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:13.161897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:13.161862    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:13.658131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:13.658095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:14.030936 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:14.030846    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"]
Nov 03 00:56:14.043667 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:14.043636    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78888f57-248nm"]
Nov 03 00:56:14.967976 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:14.967938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:15.809562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:15.809526    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1082eb74-bb4a-4295-bdcc-a3f9dca1b478" path="/var/lib/kubelet/pods/1082eb74-bb4a-4295-bdcc-a3f9dca1b478/volumes"
Nov 03 00:56:16.820653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:16.820616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:18.448587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:18.448555    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:20.107035 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:20.106990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:20.193683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:20.193646    4550 scope.go:122] "RemoveContainer" containerID="d46f7b5298d8a94ff72aee7391cf44390e262eed7f83cb419a390a4338a3ca4b"
Nov 03 00:56:21.758855 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:21.758823    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:21.875345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:21.875302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:21.964968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:21.964932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:23.486061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:23.486010    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:24.724663 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:56:24.724625    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:56:25.033247 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:25.033164    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"]
Nov 03 00:56:25.046186 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:25.046153    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ca183-kq6tm"]
Nov 03 00:56:25.240369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:25.240331    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:25.809281 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:25.809245    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135d6a8a-4dfa-4aca-960e-2c41a158fd7f" path="/var/lib/kubelet/pods/135d6a8a-4dfa-4aca-960e-2c41a158fd7f/volumes"
Nov 03 00:56:26.995478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:26.995414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:28.617342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:28.617303    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:30.330967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:30.330933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:32.007059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:32.007018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:33.605886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:33.605842    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:35.351801 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:35.351758    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:37.596230 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:37.596189    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:38.063642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:38.063607    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"]
Nov 03 00:56:38.076704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:38.076672    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a3ac151-wsw2w"]
Nov 03 00:56:39.518649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:39.518606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:39.693072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:39.693034    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:39.809995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:39.809906    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ace347-b815-421f-9086-144598bbfa34" path="/var/lib/kubelet/pods/d4ace347-b815-421f-9086-144598bbfa34/volumes"
Nov 03 00:56:40.135413 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.135323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:40.470213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.470177    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"]
Nov 03 00:56:40.476395 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.476370    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.496409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.496373    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"]
Nov 03 00:56:40.506637 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.506607    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:56:40.511726 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.511699    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:56:40.529569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.529536    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"]
Nov 03 00:56:40.534054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.534027    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.559307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.559276    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtjq\" (UniqueName: \"kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.560713 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.560682    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.560899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.560881    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.582449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.576921    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"]
Nov 03 00:56:40.593623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.593592    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"]
Nov 03 00:56:40.599640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.599619    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.627807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.627778    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-056a96-openshift-default-dockercfg-fvmh5\""
Nov 03 00:56:40.635085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.635056    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"]
Nov 03 00:56:40.643191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.643156    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:40.662841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.662797    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.662851    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.662879    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.663050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.662936    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2dt5\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663050 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663026    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.663316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663065    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.663316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663137    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663214    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663301    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663360    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663413    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtjq\" (UniqueName: \"kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.663594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663462    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663499    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.663594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663541    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.663896 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.663638    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvcx\" (UniqueName: \"kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.773545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.773452    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.821771    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.821848    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.821888    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.821932    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822042    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvcx\" (UniqueName: \"kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822069    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822099    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822125    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822197    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2dt5\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822391    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.822510    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.823126    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.774614    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.823464    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.785648    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.823739    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.830469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.823962    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.874490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.866535    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.874490 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.869611    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.921042 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.921001    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.932083 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.931166    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvcx\" (UniqueName: \"kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:40.942901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.942627    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:40.945208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:40.945141    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2dt5\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:41.094982 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.094866    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:41.096409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.096371    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb45343f-smlgz\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:41.103143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.103110    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtjq\" (UniqueName: \"kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq\") pod \"maas-api-revoke-sub-2be699cb-kw4cn\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:41.171926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.171885    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:41.236333 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.236290    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:41.303573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.296383    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:41.392424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.392389    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:41.489681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.489237    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"]
Nov 03 00:56:41.573260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.573226    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:41.678856 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.668823    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"]
Nov 03 00:56:41.681007 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:56:41.679668    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6912fc0_c1e0_4bf3_a722_daa8a11d0fd0.slice/crio-f7fb2eb372ac08c9c8d7c34c07283ccf0bcb180bdda63080d89e3486abdd75fb WatchSource:0}: Error finding container f7fb2eb372ac08c9c8d7c34c07283ccf0bcb180bdda63080d89e3486abdd75fb: Status 404 returned error can't find the container with id f7fb2eb372ac08c9c8d7c34c07283ccf0bcb180bdda63080d89e3486abdd75fb
Nov 03 00:56:41.697189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.688652    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:41.697189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.688743    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:41.697189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.688788    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:41.740353 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:56:41.740324    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab52aee_b962_4efa_8c44_6928bd93dbba.slice/crio-a5581d9a46f5d40565d4c075e23127b471bd71af3e94f180cf9f3733166f1db5 WatchSource:0}: Error finding container a5581d9a46f5d40565d4c075e23127b471bd71af3e94f180cf9f3733166f1db5: Status 404 returned error can't find the container with id a5581d9a46f5d40565d4c075e23127b471bd71af3e94f180cf9f3733166f1db5
Nov 03 00:56:41.742461 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.742413    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"]
Nov 03 00:56:41.918454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.915212    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn" event={"ID":"fab52aee-b962-4efa-8c44-6928bd93dbba","Type":"ContainerStarted","Data":"a5581d9a46f5d40565d4c075e23127b471bd71af3e94f180cf9f3733166f1db5"}
Nov 03 00:56:41.918454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.917558    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz" event={"ID":"23d78242-6d86-407a-b31c-ea2fcd3e61a6","Type":"ContainerStarted","Data":"895afb8c0a3234e30fa512d53b88fcfadb054c76ab95dbba57021a6eff05a2e5"}
Nov 03 00:56:41.921497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.920255    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" event={"ID":"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0","Type":"ContainerStarted","Data":"0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"}
Nov 03 00:56:41.921497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.920309    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" event={"ID":"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0","Type":"ContainerStarted","Data":"f7fb2eb372ac08c9c8d7c34c07283ccf0bcb180bdda63080d89e3486abdd75fb"}
Nov 03 00:56:41.969841 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:41.969177    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podStartSLOduration=1.9691548399999999 podStartE2EDuration="1.96915484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:41.964344226 +0000 UTC m=+3084.285881046" watchObservedRunningTime="2026-11-03 00:56:41.96915484 +0000 UTC m=+3084.290691641"
Nov 03 00:56:42.237384 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.237342    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:42.239394 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.239363    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:56:42.239555 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.239425    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:56:42.302359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.302309    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:42.930561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.930453    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn" event={"ID":"fab52aee-b962-4efa-8c44-6928bd93dbba","Type":"ContainerStarted","Data":"c25d9923d752902e8b290087e8d4467a3f6841b206399c4317990f0306278a44"}
Nov 03 00:56:42.932359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.932323    4550 generic.go:356] "Generic (PLEG): container finished" podID="23d78242-6d86-407a-b31c-ea2fcd3e61a6" containerID="3c2bc97f627ec6d1998698a21f2d0ec2cb848a1df2eccbea3b6a52f6aa32d36b" exitCode=0
Nov 03 00:56:42.932550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.932407    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz" event={"ID":"23d78242-6d86-407a-b31c-ea2fcd3e61a6","Type":"ContainerDied","Data":"3c2bc97f627ec6d1998698a21f2d0ec2cb848a1df2eccbea3b6a52f6aa32d36b"}
Nov 03 00:56:42.975316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:42.975268    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn" podStartSLOduration=2.975251929 podStartE2EDuration="2.975251929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:42.970682188 +0000 UTC m=+3085.292219007" watchObservedRunningTime="2026-11-03 00:56:42.975251929 +0000 UTC m=+3085.296788763"
Nov 03 00:56:43.244445 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:43.244347    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:43.318053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:43.318015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:43.943730 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:43.943694    4550 generic.go:356] "Generic (PLEG): container finished" podID="fab52aee-b962-4efa-8c44-6928bd93dbba" containerID="c25d9923d752902e8b290087e8d4467a3f6841b206399c4317990f0306278a44" exitCode=0
Nov 03 00:56:43.944216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:43.943775    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn" event={"ID":"fab52aee-b962-4efa-8c44-6928bd93dbba","Type":"ContainerDied","Data":"c25d9923d752902e8b290087e8d4467a3f6841b206399c4317990f0306278a44"}
Nov 03 00:56:43.944659 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:43.944220    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:43.945254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:43.945235    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:56:44.138835 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.138796    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:44.216622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.216518    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca\") pod \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") "
Nov 03 00:56:44.216821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.216644    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx\" (UniqueName: \"kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx\") pod \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") "
Nov 03 00:56:44.216821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.216691    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca\") pod \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\" (UID: \"23d78242-6d86-407a-b31c-ea2fcd3e61a6\") "
Nov 03 00:56:44.217024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.216985    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca" pod "23d78242-6d86-407a-b31c-ea2fcd3e61a6" (UID: "23d78242-6d86-407a-b31c-ea2fcd3e61a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:56:44.217405 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.217315    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:44.219188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.219162    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca" pod "23d78242-6d86-407a-b31c-ea2fcd3e61a6" (UID: "23d78242-6d86-407a-b31c-ea2fcd3e61a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:56:44.219937 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.219570    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx" pod "23d78242-6d86-407a-b31c-ea2fcd3e61a6" (UID: "23d78242-6d86-407a-b31c-ea2fcd3e61a6"). InnerVolumeSpecName "kube-api-access-8bvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:56:44.319526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.319488    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d78242-6d86-407a-b31c-ea2fcd3e61a6-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:44.319747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.319537    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvcx\" (UniqueName: \"kubernetes.io/projected/23d78242-6d86-407a-b31c-ea2fcd3e61a6-kube-api-access-8bvcx\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:44.322068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.322013    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:44.431519 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.431477    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:44.952117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.952084    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"
Nov 03 00:56:44.952117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.952107    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz" event={"ID":"23d78242-6d86-407a-b31c-ea2fcd3e61a6","Type":"ContainerDied","Data":"895afb8c0a3234e30fa512d53b88fcfadb054c76ab95dbba57021a6eff05a2e5"}
Nov 03 00:56:44.952653 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:44.952166    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="895afb8c0a3234e30fa512d53b88fcfadb054c76ab95dbba57021a6eff05a2e5"
Nov 03 00:56:45.153720 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.153694    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:45.237828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.237287    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca\") pod \"fab52aee-b962-4efa-8c44-6928bd93dbba\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") "
Nov 03 00:56:45.237828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.237571    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca\") pod \"fab52aee-b962-4efa-8c44-6928bd93dbba\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") "
Nov 03 00:56:45.237828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.237658    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq\" (UniqueName: \"kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq\") pod \"fab52aee-b962-4efa-8c44-6928bd93dbba\" (UID: \"fab52aee-b962-4efa-8c44-6928bd93dbba\") "
Nov 03 00:56:45.240606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.240572    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca" pod "fab52aee-b962-4efa-8c44-6928bd93dbba" (UID: "fab52aee-b962-4efa-8c44-6928bd93dbba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:56:45.241761 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.241728    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq" pod "fab52aee-b962-4efa-8c44-6928bd93dbba" (UID: "fab52aee-b962-4efa-8c44-6928bd93dbba"). InnerVolumeSpecName "kube-api-access-4qtjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:56:45.244492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.244412    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca" pod "fab52aee-b962-4efa-8c44-6928bd93dbba" (UID: "fab52aee-b962-4efa-8c44-6928bd93dbba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:56:45.341260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.341211    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:45.341512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.341284    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fab52aee-b962-4efa-8c44-6928bd93dbba-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:45.341512 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.341302    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtjq\" (UniqueName: \"kubernetes.io/projected/fab52aee-b962-4efa-8c44-6928bd93dbba-kube-api-access-4qtjq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:45.353370 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.353333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:45.965843 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.965813    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"
Nov 03 00:56:45.966523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.966493    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn" event={"ID":"fab52aee-b962-4efa-8c44-6928bd93dbba","Type":"ContainerDied","Data":"a5581d9a46f5d40565d4c075e23127b471bd71af3e94f180cf9f3733166f1db5"}
Nov 03 00:56:45.966699 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:45.966684    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5581d9a46f5d40565d4c075e23127b471bd71af3e94f180cf9f3733166f1db5"
Nov 03 00:56:46.008481 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.008426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:46.114318 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.114195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:46.176780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.176741    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"]
Nov 03 00:56:46.177477 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.177459    4550 cpu_manager.go:414] "containerMap: removing container" podUID="23d78242-6d86-407a-b31c-ea2fcd3e61a6" containerName="revoke-keys"
Nov 03 00:56:46.177564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.177478    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d78242-6d86-407a-b31c-ea2fcd3e61a6" containerName="revoke-keys"
Nov 03 00:56:46.177564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.177503    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fab52aee-b962-4efa-8c44-6928bd93dbba" containerName="revoke-keys"
Nov 03 00:56:46.177564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.177508    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab52aee-b962-4efa-8c44-6928bd93dbba" containerName="revoke-keys"
Nov 03 00:56:46.177669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.177624    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d78242-6d86-407a-b31c-ea2fcd3e61a6" containerName="revoke-keys"
Nov 03 00:56:46.177669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.177636    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab52aee-b962-4efa-8c44-6928bd93dbba" containerName="revoke-keys"
Nov 03 00:56:46.181584 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.181558    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.196978 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.196948    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-056a96\""
Nov 03 00:56:46.200579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.200551    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"]
Nov 03 00:56:46.207655 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.207629    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-056a96-dockercfg-jfjcp\""
Nov 03 00:56:46.327956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.323816    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr\") pod \"payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.327956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.324121    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.474457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.473419    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr\") pod \"payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.474457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.473696    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.569505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.569146    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.596487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.596327    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr\") pod \"payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:46.800919 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:46.800818    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:47.109047 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:56:47.109018    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cf8a78_fe36_421d_aed5_a53226509f6e.slice/crio-fdb33b69469f012c46109d4edd50e0182eeea55f33222aa2006f2932e92979f6 WatchSource:0}: Error finding container fdb33b69469f012c46109d4edd50e0182eeea55f33222aa2006f2932e92979f6: Status 404 returned error can't find the container with id fdb33b69469f012c46109d4edd50e0182eeea55f33222aa2006f2932e92979f6
Nov 03 00:56:47.111010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:47.110986    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"]
Nov 03 00:56:47.132670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:47.132637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:47.984897 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:47.984864    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" event={"ID":"69cf8a78-fe36-421d-aed5-a53226509f6e","Type":"ContainerStarted","Data":"9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e"}
Nov 03 00:56:47.985127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:47.984921    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" event={"ID":"69cf8a78-fe36-421d-aed5-a53226509f6e","Type":"ContainerStarted","Data":"fdb33b69469f012c46109d4edd50e0182eeea55f33222aa2006f2932e92979f6"}
Nov 03 00:56:47.985127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:47.984996    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:48.031094 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:48.031021    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" podStartSLOduration=2.031000045 podStartE2EDuration="2.031000045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:48.025142759 +0000 UTC m=+3090.346679558" watchObservedRunningTime="2026-11-03 00:56:48.031000045 +0000 UTC m=+3090.352536844"
Nov 03 00:56:48.993181 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:48.993144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:50.627149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:50.627111    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:51.296061 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.296018    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"]
Nov 03 00:56:51.300802 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.300771    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.306775 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.306747    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"]
Nov 03 00:56:51.311237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.311211    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.316185 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.316161    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:56:51.316369 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.316351    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:56:51.318754 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.318688    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"]
Nov 03 00:56:51.347905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.347875    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"]
Nov 03 00:56:51.472076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.472028    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.472298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.472150    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqns\" (UniqueName: \"kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.472298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.472199    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.472298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.472231    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.472521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.472505    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.472606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.472589    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxblp\" (UniqueName: \"kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.574359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.574254    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.574359 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.574353    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxblp\" (UniqueName: \"kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.574628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.574469    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.574628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.574523    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqns\" (UniqueName: \"kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.574628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.574565    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.574628 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.574604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.575130 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.575101    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.575550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.575525    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.577665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.577633    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.577665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.577652    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.595633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.595561    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqns\" (UniqueName: \"kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns\") pod \"maas-api-revoke-sub-4f6dc7ab-76jjt\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.596063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.596030    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxblp\" (UniqueName: \"kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp\") pod \"maas-api-revoke-sub-007417d6-rf6ck\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.615780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.615741    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:51.631260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.631220    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:51.932606 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:56:51.932575    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbca61a3_4d5b_4854_91c1_1765a16dd353.slice/crio-ec978f5819d63f771a0dfffc1c2e7a1f8694da5b3ba6626680a93c5a22ee3995 WatchSource:0}: Error finding container ec978f5819d63f771a0dfffc1c2e7a1f8694da5b3ba6626680a93c5a22ee3995: Status 404 returned error can't find the container with id ec978f5819d63f771a0dfffc1c2e7a1f8694da5b3ba6626680a93c5a22ee3995
Nov 03 00:56:51.935229 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.935205    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"]
Nov 03 00:56:51.969366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:51.969333    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"]
Nov 03 00:56:52.019995 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:52.019952    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt" event={"ID":"dbca61a3-4d5b-4854-91c1-1765a16dd353","Type":"ContainerStarted","Data":"ec978f5819d63f771a0dfffc1c2e7a1f8694da5b3ba6626680a93c5a22ee3995"}
Nov 03 00:56:52.021334 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:52.021300    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck" event={"ID":"d2612884-ed36-4883-88e7-b7e0ead7baa4","Type":"ContainerStarted","Data":"3541680a500cb84aec84e0e27f14fb6e72b7e58ad91d1b5607173c051d1070eb"}
Nov 03 00:56:52.345043 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:52.345004    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:53.031777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:53.031729    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt" event={"ID":"dbca61a3-4d5b-4854-91c1-1765a16dd353","Type":"ContainerStarted","Data":"a8b6d1d92852aa8e0a8f4deee59edfa036582fc13c5d17d6565efde604266492"}
Nov 03 00:56:53.033612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:53.033581    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck" event={"ID":"d2612884-ed36-4883-88e7-b7e0ead7baa4","Type":"ContainerStarted","Data":"54933def303535fd0ac260c924a0e52fb045d971b8bd5b297e4c3b2aae726dd9"}
Nov 03 00:56:53.073470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:53.073392    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt" podStartSLOduration=2.073375778 podStartE2EDuration="2.073375778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:53.071678111 +0000 UTC m=+3095.393214913" watchObservedRunningTime="2026-11-03 00:56:53.073375778 +0000 UTC m=+3095.394912623"
Nov 03 00:56:53.113889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:53.113836    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck" podStartSLOduration=2.113823409 podStartE2EDuration="2.113823409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:53.111220859 +0000 UTC m=+3095.432757657" watchObservedRunningTime="2026-11-03 00:56:53.113823409 +0000 UTC m=+3095.435360209"
Nov 03 00:56:54.024342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.024308    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:54.042625 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.042592    4550 generic.go:356] "Generic (PLEG): container finished" podID="dbca61a3-4d5b-4854-91c1-1765a16dd353" containerID="a8b6d1d92852aa8e0a8f4deee59edfa036582fc13c5d17d6565efde604266492" exitCode=0
Nov 03 00:56:54.043053 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.042679    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt" event={"ID":"dbca61a3-4d5b-4854-91c1-1765a16dd353","Type":"ContainerDied","Data":"a8b6d1d92852aa8e0a8f4deee59edfa036582fc13c5d17d6565efde604266492"}
Nov 03 00:56:54.044482 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.044460    4550 generic.go:356] "Generic (PLEG): container finished" podID="d2612884-ed36-4883-88e7-b7e0ead7baa4" containerID="54933def303535fd0ac260c924a0e52fb045d971b8bd5b297e4c3b2aae726dd9" exitCode=0
Nov 03 00:56:54.044587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.044530    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck" event={"ID":"d2612884-ed36-4883-88e7-b7e0ead7baa4","Type":"ContainerDied","Data":"54933def303535fd0ac260c924a0e52fb045d971b8bd5b297e4c3b2aae726dd9"}
Nov 03 00:56:54.725110 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:56:54.725074    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:56:54.818378 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.818344    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"]
Nov 03 00:56:54.818583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:54.818558    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" containerID="cri-o://26c205cf9cbe435ab161f9ad6c4d3932697395b02dd6e1fd8c51f1105a2843fc" gracePeriod=30
Nov 03 00:56:55.252099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.252072    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:55.301189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.301143    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:55.346051 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.345957    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns\" (UniqueName: \"kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns\") pod \"dbca61a3-4d5b-4854-91c1-1765a16dd353\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") "
Nov 03 00:56:55.346254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.346069    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca\") pod \"d2612884-ed36-4883-88e7-b7e0ead7baa4\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") "
Nov 03 00:56:55.346254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.346116    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp\" (UniqueName: \"kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp\") pod \"d2612884-ed36-4883-88e7-b7e0ead7baa4\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") "
Nov 03 00:56:55.346254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.346165    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca\") pod \"d2612884-ed36-4883-88e7-b7e0ead7baa4\" (UID: \"d2612884-ed36-4883-88e7-b7e0ead7baa4\") "
Nov 03 00:56:55.346254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.346234    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca\") pod \"dbca61a3-4d5b-4854-91c1-1765a16dd353\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") "
Nov 03 00:56:55.346544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.346351    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca\") pod \"dbca61a3-4d5b-4854-91c1-1765a16dd353\" (UID: \"dbca61a3-4d5b-4854-91c1-1765a16dd353\") "
Nov 03 00:56:55.347541 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.347510    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca" pod "dbca61a3-4d5b-4854-91c1-1765a16dd353" (UID: "dbca61a3-4d5b-4854-91c1-1765a16dd353"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:56:55.348248 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.348200    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca" pod "d2612884-ed36-4883-88e7-b7e0ead7baa4" (UID: "d2612884-ed36-4883-88e7-b7e0ead7baa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:56:55.350054 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.350030    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp" pod "d2612884-ed36-4883-88e7-b7e0ead7baa4" (UID: "d2612884-ed36-4883-88e7-b7e0ead7baa4"). InnerVolumeSpecName "kube-api-access-dxblp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:56:55.350162 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.350091    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca" pod "d2612884-ed36-4883-88e7-b7e0ead7baa4" (UID: "d2612884-ed36-4883-88e7-b7e0ead7baa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:56:55.350587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.350563    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns" pod "dbca61a3-4d5b-4854-91c1-1765a16dd353" (UID: "dbca61a3-4d5b-4854-91c1-1765a16dd353"). InnerVolumeSpecName "kube-api-access-zzqns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:56:55.350818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.350792    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca" pod "dbca61a3-4d5b-4854-91c1-1765a16dd353" (UID: "dbca61a3-4d5b-4854-91c1-1765a16dd353"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:56:55.448099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.448056    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqns\" (UniqueName: \"kubernetes.io/projected/dbca61a3-4d5b-4854-91c1-1765a16dd353-kube-api-access-zzqns\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:55.448099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.448095    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:55.448299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.448115    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxblp\" (UniqueName: \"kubernetes.io/projected/d2612884-ed36-4883-88e7-b7e0ead7baa4-kube-api-access-dxblp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:55.448299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.448125    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2612884-ed36-4883-88e7-b7e0ead7baa4-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:55.448299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.448134    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:55.448299 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.448141    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbca61a3-4d5b-4854-91c1-1765a16dd353-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:56:55.643165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:55.643082    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:56.029448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.029404    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"]
Nov 03 00:56:56.042944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.042912    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea2c7dda-jkkjb"]
Nov 03 00:56:56.062671 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.062640    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"
Nov 03 00:56:56.062848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.062670    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt" event={"ID":"dbca61a3-4d5b-4854-91c1-1765a16dd353","Type":"ContainerDied","Data":"ec978f5819d63f771a0dfffc1c2e7a1f8694da5b3ba6626680a93c5a22ee3995"}
Nov 03 00:56:56.062848 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.062757    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec978f5819d63f771a0dfffc1c2e7a1f8694da5b3ba6626680a93c5a22ee3995"
Nov 03 00:56:56.064484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.064459    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"
Nov 03 00:56:56.064484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.064475    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck" event={"ID":"d2612884-ed36-4883-88e7-b7e0ead7baa4","Type":"ContainerDied","Data":"3541680a500cb84aec84e0e27f14fb6e72b7e58ad91d1b5607173c051d1070eb"}
Nov 03 00:56:56.064669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:56.064519    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3541680a500cb84aec84e0e27f14fb6e72b7e58ad91d1b5607173c051d1070eb"
Nov 03 00:56:57.439527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:57.439490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:57.811264 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:57.811184    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9372f1-08ac-4d58-ad29-a0759f1dea35" path="/var/lib/kubelet/pods/7c9372f1-08ac-4d58-ad29-a0759f1dea35/volumes"
Nov 03 00:56:58.328592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:58.328498    4550 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:56:58.328592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:58.328575    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:56:58.999277 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:58.999241    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 00:56:59.047886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.047849    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:56:59.109168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.109133    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"]
Nov 03 00:56:59.109861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.109842    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d2612884-ed36-4883-88e7-b7e0ead7baa4" containerName="revoke-keys"
Nov 03 00:56:59.109861 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.109861    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2612884-ed36-4883-88e7-b7e0ead7baa4" containerName="revoke-keys"
Nov 03 00:56:59.109989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.109896    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dbca61a3-4d5b-4854-91c1-1765a16dd353" containerName="revoke-keys"
Nov 03 00:56:59.109989 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.109901    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbca61a3-4d5b-4854-91c1-1765a16dd353" containerName="revoke-keys"
Nov 03 00:56:59.110102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.110007    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbca61a3-4d5b-4854-91c1-1765a16dd353" containerName="revoke-keys"
Nov 03 00:56:59.110102 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.110022    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2612884-ed36-4883-88e7-b7e0ead7baa4" containerName="revoke-keys"
Nov 03 00:56:59.114824 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.114804    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.130895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.130868    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-726205-openshift-default-dockercfg-nzntw\""
Nov 03 00:56:59.137939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.137905    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"]
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225569    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225630    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225702    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrpt\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225770    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225835    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225908    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.225982    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.226052    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.231494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.226175    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.327888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.327787    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.327924    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.327998    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328067    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328186    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328280 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328273    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328311    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328335    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328352    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328271    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328381    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrpt\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328460    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328708 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328678    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.328923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.328903    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.330732 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.330712    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.330826 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.330769    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.348747 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.348713    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.348866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.348851    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrpt\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt\") pod \"e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") " pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.430027 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.429944    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:56:59.751950 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:56:59.751920    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b34cfb_64e8_4c33_8a72_bc447bab068d.slice/crio-fb5cb43c7b9e46144f0dc731da3e1b39dd5599ec5ad80c020063601f977a4ff5 WatchSource:0}: Error finding container fb5cb43c7b9e46144f0dc731da3e1b39dd5599ec5ad80c020063601f977a4ff5: Status 404 returned error can't find the container with id fb5cb43c7b9e46144f0dc731da3e1b39dd5599ec5ad80c020063601f977a4ff5
Nov 03 00:56:59.753183 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.753157    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"]
Nov 03 00:56:59.755499 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.755463    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:59.755620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.755534    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:56:59.755620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:56:59.755569    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:00.129771 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.129711    4550 generic.go:356] "Generic (PLEG): container finished" podID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerID="26c205cf9cbe435ab161f9ad6c4d3932697395b02dd6e1fd8c51f1105a2843fc" exitCode=0
Nov 03 00:57:00.130386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.130342    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" event={"ID":"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4","Type":"ContainerDied","Data":"26c205cf9cbe435ab161f9ad6c4d3932697395b02dd6e1fd8c51f1105a2843fc"}
Nov 03 00:57:00.134199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.132988    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" event={"ID":"e0b34cfb-64e8-4c33-8a72-bc447bab068d","Type":"ContainerStarted","Data":"38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc"}
Nov 03 00:57:00.134199 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.133039    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" event={"ID":"e0b34cfb-64e8-4c33-8a72-bc447bab068d","Type":"ContainerStarted","Data":"fb5cb43c7b9e46144f0dc731da3e1b39dd5599ec5ad80c020063601f977a4ff5"}
Nov 03 00:57:00.149945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.149922    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:57:00.184608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.184547    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podStartSLOduration=1.184530737 podStartE2EDuration="1.184530737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:00.179492799 +0000 UTC m=+3102.501029621" watchObservedRunningTime="2026-11-03 00:57:00.184530737 +0000 UTC m=+3102.506067536"
Nov 03 00:57:00.242226 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242192    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242304    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242469    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242558    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242691 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242675    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242711    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:00.242757 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242726    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242762    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242842    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.242971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.242882    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo\") pod \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\" (UID: \"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4\") "
Nov 03 00:57:00.243076 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243052    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:00.243356 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243335    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:00.243559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243540    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:00.243753 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243735    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.244081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243760    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.244081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243777    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.244081 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.243791    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.244926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.244903    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:00.245860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.245825    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:00.247498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.247471    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:57:00.248096 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.248068    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:00.250316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.250283    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w" pod "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" (UID: "a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4"). InnerVolumeSpecName "kube-api-access-2rk9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:00.346651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.346560    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.346651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.346599    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.346651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.346613    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.346651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.346630    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rk9w\" (UniqueName: \"kubernetes.io/projected/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-kube-api-access-2rk9w\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.346651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.346644    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:00.430783 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.430734    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:57:00.432751 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.432716    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:00.432895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.432782    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:00.682457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:00.682108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:01.141496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.141390    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"
Nov 03 00:57:01.141496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.141462    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj" event={"ID":"a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4","Type":"ContainerDied","Data":"a190856c71dadbfb11174892e34b18f7249e079b0de6bf1a4441afa75b48fd51"}
Nov 03 00:57:01.142004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.141526    4550 scope.go:122] "RemoveContainer" containerID="26c205cf9cbe435ab161f9ad6c4d3932697395b02dd6e1fd8c51f1105a2843fc"
Nov 03 00:57:01.222944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.222904    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"]
Nov 03 00:57:01.236178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.236145    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb4zj"]
Nov 03 00:57:01.432946 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.432913    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:01.433207 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.433182    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:01.812323 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:01.812244    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" path="/var/lib/kubelet/pods/a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4/volumes"
Nov 03 00:57:02.430472 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:02.430417    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:02.430885 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:02.430499    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:02.539794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:02.539760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:03.431179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:03.429790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:03.431179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:03.430487    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:03.431179 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:03.430565    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:03.550455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:03.548989    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:03.931455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:03.913585    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:04.160456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.150559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:04.265564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.258980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:04.437160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.436354    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:04.437160 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.436427    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:04.504663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.490164    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"]
Nov 03 00:57:04.504663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.492674    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy"
Nov 03 00:57:04.504663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.492698    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy"
Nov 03 00:57:04.504663 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.492905    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1d3d58b-f19d-4357-ba0d-f13cfca3bfd4" containerName="istio-proxy"
Nov 03 00:57:04.611511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.610718    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.630857 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.630828    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:57:04.646159 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.632593    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:57:04.730455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.723907    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9lmh\" (UniqueName: \"kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.730455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.724125    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.730455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.724177    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.825243 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.825149    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"]
Nov 03 00:57:04.828780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.826729    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.828780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.826811    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.828780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.827007    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9lmh\" (UniqueName: \"kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.828780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.828354    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.880539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.866837    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:04.880539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.875167    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:57:04.907874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.907522    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:57:04.934914 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:04.933633    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Nov 03 00:57:05.010363 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.010330    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9lmh\" (UniqueName: \"kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh\") pod \"maas-api-revoke-sub-ad75d9ee-xhxg2\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:05.032764 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.012482    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"]
Nov 03 00:57:05.064165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.059741    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.064165 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.059645    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"]
Nov 03 00:57:05.163513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.163443    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:05.198460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.187451    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wrjj\" (UniqueName: \"kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.199658 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.199621    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.199940 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.199921    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.308454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.301580    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.308454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.307829    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wrjj\" (UniqueName: \"kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.308454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.307984    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.320341 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.320275    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.338836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.338802    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:05.416456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.407589    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:57:05.423620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.423541    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Nov 03 00:57:05.437557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.431512    4550 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Nov 03 00:57:05.437557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.433208    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:05.437557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.433268    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:05.500463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.493600    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.500463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.494590    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wrjj\" (UniqueName: \"kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj\") pod \"maas-api-revoke-sub-ebe015c3-zm28z\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:05.798056 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:05.797706    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:06.230376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.230334    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"]
Nov 03 00:57:06.232616 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:57:06.232588    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6787d1d2_bf8f_4793_ab7e_dbc3964464f9.slice/crio-669e45fcdb1518d82a7ffa576f432b1e7b1499d4602cb12ce78b28c566082a06 WatchSource:0}: Error finding container 669e45fcdb1518d82a7ffa576f432b1e7b1499d4602cb12ce78b28c566082a06: Status 404 returned error can't find the container with id 669e45fcdb1518d82a7ffa576f432b1e7b1499d4602cb12ce78b28c566082a06
Nov 03 00:57:06.432780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.432555    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:06.432780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.432617    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:06.452142 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.452107    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"]
Nov 03 00:57:06.466005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.462047    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.506685 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.506647    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ac338d\""
Nov 03 00:57:06.528522 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.528491    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"openshift-service-ca.crt\""
Nov 03 00:57:06.528859 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.528838    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"payload-processing-external-model-e2e--0fa13bfc-dockercfg-v78wj\""
Nov 03 00:57:06.529058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.529044    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"agc-cred-f719de565e1c\""
Nov 03 00:57:06.529195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.529183    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"routing-overlay\""
Nov 03 00:57:06.529505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.529488    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"kube-root-ca.crt\""
Nov 03 00:57:06.550489 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.547533    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"]
Nov 03 00:57:06.555100 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.555070    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxjb\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.555342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.555318    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.555421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.555373    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.555529 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.555506    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.574939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.563285    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"]
Nov 03 00:57:06.622894 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.622858    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2" event={"ID":"6787d1d2-bf8f-4793-ab7e-dbc3964464f9","Type":"ContainerStarted","Data":"669e45fcdb1518d82a7ffa576f432b1e7b1499d4602cb12ce78b28c566082a06"}
Nov 03 00:57:06.678746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.663490    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.678746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.663787    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxjb\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.678746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.663868    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.678746 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.663922    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.680224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.679850    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.681062 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.681013    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.716156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.716076    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.728804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.728722    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxjb\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.924028 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.923994    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:06.959728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:06.948131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:07.439977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.439521    4550 patch_prober.go:29] interesting pod/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body=
Nov 03 00:57:07.439977 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.439594    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused"
Nov 03 00:57:07.636942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.636843    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z" event={"ID":"938de3a0-05f9-4c4f-8666-5f230d718a68","Type":"ContainerStarted","Data":"0375bd5af04f02265195fb22c3a074a24ada1419cc86a8295294b3a7f2900958"}
Nov 03 00:57:07.636942 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.636937    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z" event={"ID":"938de3a0-05f9-4c4f-8666-5f230d718a68","Type":"ContainerStarted","Data":"ac3b1014a5429fa3310839ff29164f245cc293ef28eb808657846d14c89b8cba"}
Nov 03 00:57:07.638651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.638627    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2" event={"ID":"6787d1d2-bf8f-4793-ab7e-dbc3964464f9","Type":"ContainerStarted","Data":"120f3c56a234db383ebff7ace076864595262709059628572f3407bcf8ccd182"}
Nov 03 00:57:07.732848 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:57:07.732241    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cd023f_281a_4638_8977_2de66ee43385.slice/crio-c28effc58e81ee7261b7733cad6fc2b4297368a2ba8e0883050c233c7382484f WatchSource:0}: Error finding container c28effc58e81ee7261b7733cad6fc2b4297368a2ba8e0883050c233c7382484f: Status 404 returned error can't find the container with id c28effc58e81ee7261b7733cad6fc2b4297368a2ba8e0883050c233c7382484f
Nov 03 00:57:07.733904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.733844    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z" podStartSLOduration=3.7338247730000003 podStartE2EDuration="3.733824773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:07.684870483 +0000 UTC m=+3110.006407284" watchObservedRunningTime="2026-11-03 00:57:07.733824773 +0000 UTC m=+3110.055361573"
Nov 03 00:57:07.739316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.739285    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"]
Nov 03 00:57:07.741554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:07.741513    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2" podStartSLOduration=3.741499866 podStartE2EDuration="3.741499866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:07.734552416 +0000 UTC m=+3110.056089240" watchObservedRunningTime="2026-11-03 00:57:07.741499866 +0000 UTC m=+3110.063036665"
Nov 03 00:57:08.521963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.521929    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:57:08.522829 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.522563    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:57:08.524150 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.524118    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 00:57:08.550330 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.550270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:08.709116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.707866    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7" event={"ID":"a4cd023f-281a-4638-8977-2de66ee43385","Type":"ContainerStarted","Data":"9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80"}
Nov 03 00:57:08.709116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.707932    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7" event={"ID":"a4cd023f-281a-4638-8977-2de66ee43385","Type":"ContainerStarted","Data":"c28effc58e81ee7261b7733cad6fc2b4297368a2ba8e0883050c233c7382484f"}
Nov 03 00:57:08.709116 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.709047    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:08.712258 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.712141    4550 generic.go:356] "Generic (PLEG): container finished" podID="938de3a0-05f9-4c4f-8666-5f230d718a68" containerID="0375bd5af04f02265195fb22c3a074a24ada1419cc86a8295294b3a7f2900958" exitCode=0
Nov 03 00:57:08.712606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.712497    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z" event={"ID":"938de3a0-05f9-4c4f-8666-5f230d718a68","Type":"ContainerDied","Data":"0375bd5af04f02265195fb22c3a074a24ada1419cc86a8295294b3a7f2900958"}
Nov 03 00:57:08.715188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.715151    4550 generic.go:356] "Generic (PLEG): container finished" podID="6787d1d2-bf8f-4793-ab7e-dbc3964464f9" containerID="120f3c56a234db383ebff7ace076864595262709059628572f3407bcf8ccd182" exitCode=0
Nov 03 00:57:08.715944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.715913    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2" event={"ID":"6787d1d2-bf8f-4793-ab7e-dbc3964464f9","Type":"ContainerDied","Data":"120f3c56a234db383ebff7ace076864595262709059628572f3407bcf8ccd182"}
Nov 03 00:57:08.784513 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:08.784214    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7" podStartSLOduration=2.7841924689999997 podStartE2EDuration="2.784192469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:08.781994207 +0000 UTC m=+3111.103531007" watchObservedRunningTime="2026-11-03 00:57:08.784192469 +0000 UTC m=+3111.105729271"
Nov 03 00:57:09.924823 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:09.924789    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:10.032447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.032410    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:10.036416 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.036388    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:10.114049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114018    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj\" (UniqueName: \"kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj\") pod \"938de3a0-05f9-4c4f-8666-5f230d718a68\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") "
Nov 03 00:57:10.114198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114101    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca\") pod \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") "
Nov 03 00:57:10.114198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114176    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca\") pod \"938de3a0-05f9-4c4f-8666-5f230d718a68\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") "
Nov 03 00:57:10.114354 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114265    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca\") pod \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") "
Nov 03 00:57:10.114414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114394    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh\" (UniqueName: \"kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh\") pod \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\" (UID: \"6787d1d2-bf8f-4793-ab7e-dbc3964464f9\") "
Nov 03 00:57:10.114495 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114459    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca\") pod \"938de3a0-05f9-4c4f-8666-5f230d718a68\" (UID: \"938de3a0-05f9-4c4f-8666-5f230d718a68\") "
Nov 03 00:57:10.114599 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.114572    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca" pod "6787d1d2-bf8f-4793-ab7e-dbc3964464f9" (UID: "6787d1d2-bf8f-4793-ab7e-dbc3964464f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:10.115401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.115374    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:10.119401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.117130    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca" pod "938de3a0-05f9-4c4f-8666-5f230d718a68" (UID: "938de3a0-05f9-4c4f-8666-5f230d718a68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:10.119401 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.118203    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca" pod "938de3a0-05f9-4c4f-8666-5f230d718a68" (UID: "938de3a0-05f9-4c4f-8666-5f230d718a68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:10.128700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.121065    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh" pod "6787d1d2-bf8f-4793-ab7e-dbc3964464f9" (UID: "6787d1d2-bf8f-4793-ab7e-dbc3964464f9"). InnerVolumeSpecName "kube-api-access-r9lmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:10.128700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.127011    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj" pod "938de3a0-05f9-4c4f-8666-5f230d718a68" (UID: "938de3a0-05f9-4c4f-8666-5f230d718a68"). InnerVolumeSpecName "kube-api-access-4wrjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:10.128700 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.128263    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca" pod "6787d1d2-bf8f-4793-ab7e-dbc3964464f9" (UID: "6787d1d2-bf8f-4793-ab7e-dbc3964464f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:10.137204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.137040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:10.236972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.216860    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:10.236972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.216896    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9lmh\" (UniqueName: \"kubernetes.io/projected/6787d1d2-bf8f-4793-ab7e-dbc3964464f9-kube-api-access-r9lmh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:10.236972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.216911    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:10.236972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.216927    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wrjj\" (UniqueName: \"kubernetes.io/projected/938de3a0-05f9-4c4f-8666-5f230d718a68-kube-api-access-4wrjj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:10.236972 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.216940    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938de3a0-05f9-4c4f-8666-5f230d718a68-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:10.732501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.732468    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"
Nov 03 00:57:10.732501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.732487    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z" event={"ID":"938de3a0-05f9-4c4f-8666-5f230d718a68","Type":"ContainerDied","Data":"ac3b1014a5429fa3310839ff29164f245cc293ef28eb808657846d14c89b8cba"}
Nov 03 00:57:10.732889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.732547    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac3b1014a5429fa3310839ff29164f245cc293ef28eb808657846d14c89b8cba"
Nov 03 00:57:10.734171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.734144    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"
Nov 03 00:57:10.734327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.734172    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2" event={"ID":"6787d1d2-bf8f-4793-ab7e-dbc3964464f9","Type":"ContainerDied","Data":"669e45fcdb1518d82a7ffa576f432b1e7b1499d4602cb12ce78b28c566082a06"}
Nov 03 00:57:10.734327 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:10.734213    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="669e45fcdb1518d82a7ffa576f432b1e7b1499d4602cb12ce78b28c566082a06"
Nov 03 00:57:11.742922 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:11.742887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:13.414695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:13.414659    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:13.763589 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:13.763496    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:14.777752 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.777717    4550 generic.go:356] "Generic (PLEG): container finished" podID="e203d55a-e393-443b-b9f3-ac898417f112" containerID="2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a" exitCode=0
Nov 03 00:57:14.778296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.777792    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" event={"ID":"e203d55a-e393-443b-b9f3-ac898417f112","Type":"ContainerDied","Data":"2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"}
Nov 03 00:57:14.778296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.778275    4550 scope.go:122] "RemoveContainer" containerID="2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"
Nov 03 00:57:14.780459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.780400    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:14.780579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.780485    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:14.780579 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.780520    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:14.992617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:14.992528    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:57:15.041423 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.041384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:15.087815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.087779    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"]
Nov 03 00:57:15.089351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.088864    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6787d1d2-bf8f-4793-ab7e-dbc3964464f9" containerName="revoke-keys"
Nov 03 00:57:15.089351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.088891    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6787d1d2-bf8f-4793-ab7e-dbc3964464f9" containerName="revoke-keys"
Nov 03 00:57:15.089351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.088957    4550 cpu_manager.go:414] "containerMap: removing container" podUID="938de3a0-05f9-4c4f-8666-5f230d718a68" containerName="revoke-keys"
Nov 03 00:57:15.089351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.088966    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938de3a0-05f9-4c4f-8666-5f230d718a68" containerName="revoke-keys"
Nov 03 00:57:15.089351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.089107    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="938de3a0-05f9-4c4f-8666-5f230d718a68" containerName="revoke-keys"
Nov 03 00:57:15.089351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.089125    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6787d1d2-bf8f-4793-ab7e-dbc3964464f9" containerName="revoke-keys"
Nov 03 00:57:15.114703 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.113327    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.133451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.130493    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:57:15.133451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.130657    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:57:15.145450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.144502    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"]
Nov 03 00:57:15.218279 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.218229    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.218564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.218405    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh5qj\" (UniqueName: \"kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.218564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.218543    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.320244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.320156    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.320453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.320296    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh5qj\" (UniqueName: \"kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.320453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.320383    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.321030 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.321000    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.323244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.323219    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.340630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.340601    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh5qj\" (UniqueName: \"kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj\") pod \"maas-api-revoke-sub-37f890fc-5wwqb\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.455557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.455519    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:15.566455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.564656    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"]
Nov 03 00:57:15.591501 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.584626    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.610678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.609828    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"]
Nov 03 00:57:15.625412 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.625368    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.626748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.625647    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.626748 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.625731    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5grp\" (UniqueName: \"kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.727551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.727515    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.727735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.727624    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.727735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.727693    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5grp\" (UniqueName: \"kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.728383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.728357    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.730013 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.729991    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.779454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.755519    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5grp\" (UniqueName: \"kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp\") pod \"maas-api-revoke-sub-aacf581f-6q62v\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:15.803325 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:57:15.803286    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95ce9ef_83ec_42e9_ba3b_825d05c507cf.slice/crio-aae5bc6baddef4dcd3a417f4a41759a446b09364970550436b8cb6c3dc0201d7 WatchSource:0}: Error finding container aae5bc6baddef4dcd3a417f4a41759a446b09364970550436b8cb6c3dc0201d7: Status 404 returned error can't find the container with id aae5bc6baddef4dcd3a417f4a41759a446b09364970550436b8cb6c3dc0201d7
Nov 03 00:57:15.821727 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.821698    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"]
Nov 03 00:57:15.821867 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.821775    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" event={"ID":"e203d55a-e393-443b-b9f3-ac898417f112","Type":"ContainerStarted","Data":"03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b"}
Nov 03 00:57:15.918262 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:15.918226    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:16.293214 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.293185    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"]
Nov 03 00:57:16.552558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.552524    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:57:16.554288 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.554258    4550 patch_prober.go:29] interesting pod/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:57:16.554478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.554317    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 00:57:16.900551 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.900507    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v" event={"ID":"babea784-bbcc-4f41-8284-9dcb22ab3539","Type":"ContainerStarted","Data":"8cbd40f878c049c7545909a04cddf68ce294191d52cf90609847e5b468eabd6f"}
Nov 03 00:57:16.901178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.901151    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v" event={"ID":"babea784-bbcc-4f41-8284-9dcb22ab3539","Type":"ContainerStarted","Data":"17c14ae7920b8815aa7f87578127e277cf3c7a6c80947d3f654a166ec413bc62"}
Nov 03 00:57:16.917662 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.917625    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb" event={"ID":"d95ce9ef-83ec-42e9-ba3b-825d05c507cf","Type":"ContainerStarted","Data":"cc4e997a2cffe28216c4d77d3ae685c272a1ea34ecfc928b49c8d1f5a5407233"}
Nov 03 00:57:16.924556 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:16.924511    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb" event={"ID":"d95ce9ef-83ec-42e9-ba3b-825d05c507cf","Type":"ContainerStarted","Data":"aae5bc6baddef4dcd3a417f4a41759a446b09364970550436b8cb6c3dc0201d7"}
Nov 03 00:57:17.000316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.000279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:17.039520 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.036057    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v" podStartSLOduration=2.036036949 podStartE2EDuration="2.036036949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:16.989129778 +0000 UTC m=+3119.310666579" watchObservedRunningTime="2026-11-03 00:57:17.036036949 +0000 UTC m=+3119.357573750"
Nov 03 00:57:17.078459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.077631    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb" podStartSLOduration=2.077605583 podStartE2EDuration="2.077605583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:17.033677407 +0000 UTC m=+3119.355214207" watchObservedRunningTime="2026-11-03 00:57:17.077605583 +0000 UTC m=+3119.399142388"
Nov 03 00:57:17.553000 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.552961    4550 patch_prober.go:29] interesting pod/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:57:17.553208 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.553025    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 00:57:17.936067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.936033    4550 generic.go:356] "Generic (PLEG): container finished" podID="d95ce9ef-83ec-42e9-ba3b-825d05c507cf" containerID="cc4e997a2cffe28216c4d77d3ae685c272a1ea34ecfc928b49c8d1f5a5407233" exitCode=0
Nov 03 00:57:17.936640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.936117    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb" event={"ID":"d95ce9ef-83ec-42e9-ba3b-825d05c507cf","Type":"ContainerDied","Data":"cc4e997a2cffe28216c4d77d3ae685c272a1ea34ecfc928b49c8d1f5a5407233"}
Nov 03 00:57:17.940721 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.940690    4550 generic.go:356] "Generic (PLEG): container finished" podID="babea784-bbcc-4f41-8284-9dcb22ab3539" containerID="8cbd40f878c049c7545909a04cddf68ce294191d52cf90609847e5b468eabd6f" exitCode=0
Nov 03 00:57:17.941886 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:17.941855    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v" event={"ID":"babea784-bbcc-4f41-8284-9dcb22ab3539","Type":"ContainerDied","Data":"8cbd40f878c049c7545909a04cddf68ce294191d52cf90609847e5b468eabd6f"}
Nov 03 00:57:18.553136 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:18.553096    4550 patch_prober.go:29] interesting pod/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:57:18.553365 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:18.553155    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 00:57:18.802022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:18.801988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:18.902462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:18.902398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:19.345918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.345879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:19.386467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.385694    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:19.393080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.393053    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:19.450254 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.450158    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp\" (UniqueName: \"kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp\") pod \"babea784-bbcc-4f41-8284-9dcb22ab3539\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") "
Nov 03 00:57:19.450475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.450254    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca\") pod \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") "
Nov 03 00:57:19.450475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.450347    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj\" (UniqueName: \"kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj\") pod \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") "
Nov 03 00:57:19.450475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.450461    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca\") pod \"babea784-bbcc-4f41-8284-9dcb22ab3539\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") "
Nov 03 00:57:19.450674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.450555    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca\") pod \"babea784-bbcc-4f41-8284-9dcb22ab3539\" (UID: \"babea784-bbcc-4f41-8284-9dcb22ab3539\") "
Nov 03 00:57:19.450814 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.450792    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca\") pod \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\" (UID: \"d95ce9ef-83ec-42e9-ba3b-825d05c507cf\") "
Nov 03 00:57:19.451108 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.451069    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca" pod "d95ce9ef-83ec-42e9-ba3b-825d05c507cf" (UID: "d95ce9ef-83ec-42e9-ba3b-825d05c507cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:19.451883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.451555    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:19.451883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.451612    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca" pod "babea784-bbcc-4f41-8284-9dcb22ab3539" (UID: "babea784-bbcc-4f41-8284-9dcb22ab3539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:19.453877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.453850    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj" pod "d95ce9ef-83ec-42e9-ba3b-825d05c507cf" (UID: "d95ce9ef-83ec-42e9-ba3b-825d05c507cf"). InnerVolumeSpecName "kube-api-access-wh5qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:19.454104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.454063    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca" pod "d95ce9ef-83ec-42e9-ba3b-825d05c507cf" (UID: "d95ce9ef-83ec-42e9-ba3b-825d05c507cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:19.459259 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.455618    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp" pod "babea784-bbcc-4f41-8284-9dcb22ab3539" (UID: "babea784-bbcc-4f41-8284-9dcb22ab3539"). InnerVolumeSpecName "kube-api-access-g5grp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:19.459553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.459527    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca" pod "babea784-bbcc-4f41-8284-9dcb22ab3539" (UID: "babea784-bbcc-4f41-8284-9dcb22ab3539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:57:19.552690 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.552634    4550 patch_prober.go:29] interesting pod/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:57:19.552924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.552714    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 00:57:19.553518 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.553493    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh5qj\" (UniqueName: \"kubernetes.io/projected/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-kube-api-access-wh5qj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:19.553560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.553521    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:19.553560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.553530    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babea784-bbcc-4f41-8284-9dcb22ab3539-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:19.553560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.553541    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95ce9ef-83ec-42e9-ba3b-825d05c507cf-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:19.553560 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.553550    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5grp\" (UniqueName: \"kubernetes.io/projected/babea784-bbcc-4f41-8284-9dcb22ab3539-kube-api-access-g5grp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:19.741329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.741251    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"]
Nov 03 00:57:19.742284 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.742266    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d95ce9ef-83ec-42e9-ba3b-825d05c507cf" containerName="revoke-keys"
Nov 03 00:57:19.742362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.742287    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95ce9ef-83ec-42e9-ba3b-825d05c507cf" containerName="revoke-keys"
Nov 03 00:57:19.742362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.742316    4550 cpu_manager.go:414] "containerMap: removing container" podUID="babea784-bbcc-4f41-8284-9dcb22ab3539" containerName="revoke-keys"
Nov 03 00:57:19.742362 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.742324    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babea784-bbcc-4f41-8284-9dcb22ab3539" containerName="revoke-keys"
Nov 03 00:57:19.742502 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.742490    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95ce9ef-83ec-42e9-ba3b-825d05c507cf" containerName="revoke-keys"
Nov 03 00:57:19.742543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.742507    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="babea784-bbcc-4f41-8284-9dcb22ab3539" containerName="revoke-keys"
Nov 03 00:57:19.750404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.750370    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.774796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.774765    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7f20f7-openshift-default-dockercfg-jqlmk\""
Nov 03 00:57:19.806466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.800698    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"]
Nov 03 00:57:19.836455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.826362    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.884522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.884574    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.884630    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.884742    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.884793    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.885113    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2bj\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.885199    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.885297    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.886470 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.885331    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:19.934454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:19.933529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:20.165717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.153837    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v" event={"ID":"babea784-bbcc-4f41-8284-9dcb22ab3539","Type":"ContainerDied","Data":"17c14ae7920b8815aa7f87578127e277cf3c7a6c80947d3f654a166ec413bc62"}
Nov 03 00:57:20.165717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.153899    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c14ae7920b8815aa7f87578127e277cf3c7a6c80947d3f654a166ec413bc62"
Nov 03 00:57:20.165717 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.153975    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"
Nov 03 00:57:20.195617 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.195583    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb" event={"ID":"d95ce9ef-83ec-42e9-ba3b-825d05c507cf","Type":"ContainerDied","Data":"aae5bc6baddef4dcd3a417f4a41759a446b09364970550436b8cb6c3dc0201d7"}
Nov 03 00:57:20.195900 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.195878    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aae5bc6baddef4dcd3a417f4a41759a446b09364970550436b8cb6c3dc0201d7"
Nov 03 00:57:20.196105 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.196092    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"
Nov 03 00:57:20.266677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266636    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2bj\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.266883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266729    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.266883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266803    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.266883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266834    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.266883 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266874    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.267114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266898    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.267114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.266952    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.267114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.267054    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.267114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.267109    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.286562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.276957    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.286562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.277533    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.286562 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.277770    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.290606 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.290346    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.291011 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.290926    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.308917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.308878    4550 scope.go:122] "RemoveContainer" containerID="e8423f1b475e49a9d407b03d04d5906e74bc536dc8a76decb94178a441fe7b0e"
Nov 03 00:57:20.423807 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.423544    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2bj\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.424494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.423979    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.424587 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.424510    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.426866 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.426840    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token\") pod \"e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") " pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.433641 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.433617    4550 scope.go:122] "RemoveContainer" containerID="1278e47955562051a6b84920e8ca483a30749b134930eae64addd5ee19501799"
Nov 03 00:57:20.441125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.441098    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:20.446317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.446286    4550 scope.go:122] "RemoveContainer" containerID="ab44ef6c5862e943876e6cea20b46d27bd5e4a9d01451ebe109cca0ef9b17874"
Nov 03 00:57:20.490170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.490032    4550 scope.go:122] "RemoveContainer" containerID="17ad68fee563a9f6add52e8beb2b433aef6c51f884e3592d4d8a6048939931d6"
Nov 03 00:57:20.544221 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.544156    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:20.553336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.552916    4550 patch_prober.go:29] interesting pod/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:57:20.553336 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.552972    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 00:57:20.751463 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:20.750423    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:21.084535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.081308    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"]
Nov 03 00:57:21.097455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.094172    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:21.097455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.094272    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:21.097455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.094327    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:21.210950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.210919    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" event={"ID":"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a","Type":"ContainerStarted","Data":"2ed8afbdc83f5160322ffc3d34cf1038bf39b4d82af637a4b97ae49f5104b9c7"}
Nov 03 00:57:21.553023 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.552989    4550 patch_prober.go:29] interesting pod/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:57:21.553497 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:21.553048    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Nov 03 00:57:22.219597 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.219562    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" event={"ID":"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a","Type":"ContainerStarted","Data":"6761bec57d1b47678c389457d3985f2dbc8a8aa10ea0865dab68aaf0bd194d97"}
Nov 03 00:57:22.289947 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.289883    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podStartSLOduration=3.289863988 podStartE2EDuration="3.289863988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:22.278418982 +0000 UTC m=+3124.599955784" watchObservedRunningTime="2026-11-03 00:57:22.289863988 +0000 UTC m=+3124.611400792"
Nov 03 00:57:22.329155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.329119    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:22.441962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.441928    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:22.481508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.480786    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:22.560467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.560421    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:57:22.560983 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.560952    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:57:22.561788 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:22.561762    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 00:57:23.228170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:23.228134    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:23.229149 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:23.229126    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:57:24.114917 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:24.114877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:24.604971 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:24.604933    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:24.722473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:24.721674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:24.726767 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:57:24.726732    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:57:24.869836 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:24.869724    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:24.986672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:24.986609    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"]
Nov 03 00:57:24.986968 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:24.986937    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" containerID="cri-o://051cc072ff14609abdf50970af10110034e7041c45a1336bf237049431b194f3" gracePeriod=30
Nov 03 00:57:25.019479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:25.004563    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"]
Nov 03 00:57:25.019479 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:25.004851    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" containerID="cri-o://243527cb1579e1365f23736396330d2f8a9f1372ac0c08dc73e6852638436923" gracePeriod=30
Nov 03 00:57:26.063109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:26.063066    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:26.307191 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:26.307151    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:26.407683 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:26.407653    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:26.936139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:26.936101    4550 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:57:26.936443 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:26.936402    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:57:27.077809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.077776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:27.179004 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.178883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:27.513414 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.513375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:27.633462 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.633039    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:27.720582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.720546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:27.889517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.889418    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"]
Nov 03 00:57:27.950880 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:27.950844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:28.079895 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:28.079851    4550 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:57:28.080404 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:28.079917    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:57:28.109157 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:28.109125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:28.209645 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:28.209611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:28.771736 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:28.771706    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:29.743487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:29.743454    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:29.929080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:29.929040    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:30.440559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.440519    4550 generic.go:356] "Generic (PLEG): container finished" podID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerID="243527cb1579e1365f23736396330d2f8a9f1372ac0c08dc73e6852638436923" exitCode=0
Nov 03 00:57:30.440923 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.440893    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" event={"ID":"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163","Type":"ContainerDied","Data":"243527cb1579e1365f23736396330d2f8a9f1372ac0c08dc73e6852638436923"}
Nov 03 00:57:30.445282 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.445251    4550 generic.go:356] "Generic (PLEG): container finished" podID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerID="051cc072ff14609abdf50970af10110034e7041c45a1336bf237049431b194f3" exitCode=0
Nov 03 00:57:30.445568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.445489    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" event={"ID":"5b469d88-aac1-49ed-9a16-0e375bc165e7","Type":"ContainerDied","Data":"051cc072ff14609abdf50970af10110034e7041c45a1336bf237049431b194f3"}
Nov 03 00:57:30.599926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.599843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:30.630091 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.630062    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:57:30.706695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.706619    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:57:30.809193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809154    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809289    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809325    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809369    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809418    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809536    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809565    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809607    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809630    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809639    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809721    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.809762 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809760    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809806    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809883    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809944    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.809984    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810036    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810081    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810139    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy\") pod \"5b469d88-aac1-49ed-9a16-0e375bc165e7\" (UID: \"5b469d88-aac1-49ed-9a16-0e375bc165e7\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810209    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810287    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo\") pod \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\" (UID: \"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163\") "
Nov 03 00:57:30.810376 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810309    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.810950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810521    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.810950 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.810891    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:30.813022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.812747    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.813022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.812773    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.813022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.812788    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b469d88-aac1-49ed-9a16-0e375bc165e7-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.813022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.812802    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.813022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.812820    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.813022 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.812833    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.813459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.813234    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:57:30.813523 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.813458    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.813592 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.813513    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.814002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.813657    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.814002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.813703    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.814002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.813750    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.814544 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.814499    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "kube-api-access-jwt6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:30.814890 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.814866    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:57:30.815904 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.815875    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:30.816200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.816176    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:30.816613 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.816591    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2" pod "5b469d88-aac1-49ed-9a16-0e375bc165e7" (UID: "5b469d88-aac1-49ed-9a16-0e375bc165e7"). InnerVolumeSpecName "kube-api-access-9kdv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:30.816876 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.816859    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token" pod "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" (UID: "cad9f8fd-b0cd-45ed-a8e0-89e42edbe163"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915098    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915405    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915527    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915549    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915564    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915621    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915636    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915649    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915661    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915676    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdv2\" (UniqueName: \"kubernetes.io/projected/5b469d88-aac1-49ed-9a16-0e375bc165e7-kube-api-access-9kdv2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915688    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b469d88-aac1-49ed-9a16-0e375bc165e7-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:30.920901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:30.915702    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwt6x\" (UniqueName: \"kubernetes.io/projected/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163-kube-api-access-jwt6x\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:31.455735 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.455701    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"
Nov 03 00:57:31.455955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.455731    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg" event={"ID":"5b469d88-aac1-49ed-9a16-0e375bc165e7","Type":"ContainerDied","Data":"3027a460f7e24fa283cb099a502cce6e47a9e28247bbc45096e0b67c5d0b65ed"}
Nov 03 00:57:31.455955 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.455802    4550 scope.go:122] "RemoveContainer" containerID="051cc072ff14609abdf50970af10110034e7041c45a1336bf237049431b194f3"
Nov 03 00:57:31.457684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.457656    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8" event={"ID":"cad9f8fd-b0cd-45ed-a8e0-89e42edbe163","Type":"ContainerDied","Data":"508fb434d1ba89555dcffca201259683bc008832d61e34e06c641b76d93a9365"}
Nov 03 00:57:31.457804 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.457683    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"
Nov 03 00:57:31.494375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.491678    4550 scope.go:122] "RemoveContainer" containerID="243527cb1579e1365f23736396330d2f8a9f1372ac0c08dc73e6852638436923"
Nov 03 00:57:31.558488 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.558449    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"]
Nov 03 00:57:31.571901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.571863    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-559tg"]
Nov 03 00:57:31.618608 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.618576    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"]
Nov 03 00:57:31.634564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.634530    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-rn9x8"]
Nov 03 00:57:31.712296 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.712212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:31.811633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.811591    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" path="/var/lib/kubelet/pods/5b469d88-aac1-49ed-9a16-0e375bc165e7/volumes"
Nov 03 00:57:31.812397 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:31.812375    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" path="/var/lib/kubelet/pods/cad9f8fd-b0cd-45ed-a8e0-89e42edbe163/volumes"
Nov 03 00:57:32.890997 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:32.890930    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7" podUID="a4cd023f-281a-4638-8977-2de66ee43385" containerName="payload-processing" containerID="cri-o://9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80" gracePeriod=25
Nov 03 00:57:33.199620 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.199590    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:33.282911 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.282872    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb\") pod \"a4cd023f-281a-4638-8977-2de66ee43385\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") "
Nov 03 00:57:33.283127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.282983    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume\") pod \"a4cd023f-281a-4638-8977-2de66ee43385\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") "
Nov 03 00:57:33.283127 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.283034    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials\") pod \"a4cd023f-281a-4638-8977-2de66ee43385\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") "
Nov 03 00:57:33.283249 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.283148    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume\") pod \"a4cd023f-281a-4638-8977-2de66ee43385\" (UID: \"a4cd023f-281a-4638-8977-2de66ee43385\") "
Nov 03 00:57:33.283789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.283748    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume" pod "a4cd023f-281a-4638-8977-2de66ee43385" (UID: "a4cd023f-281a-4638-8977-2de66ee43385"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:33.283789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.283769    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume" pod "a4cd023f-281a-4638-8977-2de66ee43385" (UID: "a4cd023f-281a-4638-8977-2de66ee43385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:33.284092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.284074    4550 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-routing-overlay-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:33.284171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.284095    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cd023f-281a-4638-8977-2de66ee43385-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:33.285996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.285972    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials" pod "a4cd023f-281a-4638-8977-2de66ee43385" (UID: "a4cd023f-281a-4638-8977-2de66ee43385"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:33.286117 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.286092    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb" pod "a4cd023f-281a-4638-8977-2de66ee43385" (UID: "a4cd023f-281a-4638-8977-2de66ee43385"). InnerVolumeSpecName "kube-api-access-zsxjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:33.336453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.336404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:33.386082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.386047    4550 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-provider-credentials\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:33.386082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.386077    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxjb\" (UniqueName: \"kubernetes.io/projected/a4cd023f-281a-4638-8977-2de66ee43385-kube-api-access-zsxjb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:33.480566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.480465    4550 generic.go:356] "Generic (PLEG): container finished" podID="a4cd023f-281a-4638-8977-2de66ee43385" containerID="9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80" exitCode=0
Nov 03 00:57:33.480566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.480512    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"
Nov 03 00:57:33.480566 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.480516    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7" event={"ID":"a4cd023f-281a-4638-8977-2de66ee43385","Type":"ContainerDied","Data":"9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80"}
Nov 03 00:57:33.480820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.480570    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7" event={"ID":"a4cd023f-281a-4638-8977-2de66ee43385","Type":"ContainerDied","Data":"c28effc58e81ee7261b7733cad6fc2b4297368a2ba8e0883050c233c7382484f"}
Nov 03 00:57:33.480820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.480593    4550 scope.go:122] "RemoveContainer" containerID="9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80"
Nov 03 00:57:33.495540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.494879    4550 scope.go:122] "RemoveContainer" containerID="9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80"
Nov 03 00:57:33.495540 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:57:33.495321    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80\": container with ID starting with 9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80 not found: ID does not exist" containerID="9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80"
Nov 03 00:57:33.495540 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.495365    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80"} err="failed to get container status \"9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80\": rpc error: code = NotFound desc = could not find container \"9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80\": container with ID starting with 9f0f7794277614a924b74ca366292a0c70fb9c6c77ab6ca84f0a6d3ecbd66a80 not found: ID does not exist"
Nov 03 00:57:33.543744 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.543709    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"]
Nov 03 00:57:33.548794 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.548763    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bd2mjt7"]
Nov 03 00:57:33.809383 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:33.809294    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cd023f-281a-4638-8977-2de66ee43385" path="/var/lib/kubelet/pods/a4cd023f-281a-4638-8977-2de66ee43385/volumes"
Nov 03 00:57:34.581913 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:34.581876    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:34.690234 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:34.690171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:34.950966 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:34.950923    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.043064 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.043030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.196781 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.196738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.404507 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.402328    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.602092 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.602056    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.606266 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.606234    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"]
Nov 03 00:57:35.607550 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.607524    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a4cd023f-281a-4638-8977-2de66ee43385" containerName="payload-processing"
Nov 03 00:57:35.607695 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.607680    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cd023f-281a-4638-8977-2de66ee43385" containerName="payload-processing"
Nov 03 00:57:35.607815 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.607804    4550 cpu_manager.go:414] "containerMap: removing container" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy"
Nov 03 00:57:35.607915 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.607903    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy"
Nov 03 00:57:35.608024 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.608011    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy"
Nov 03 00:57:35.608111 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.608099    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy"
Nov 03 00:57:35.608400 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.608384    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad9f8fd-b0cd-45ed-a8e0-89e42edbe163" containerName="istio-proxy"
Nov 03 00:57:35.608564 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.608548    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b469d88-aac1-49ed-9a16-0e375bc165e7" containerName="istio-proxy"
Nov 03 00:57:35.608681 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.608670    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cd023f-281a-4638-8977-2de66ee43385" containerName="payload-processing"
Nov 03 00:57:35.624005 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.623980    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.649860 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.649835    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8ed55a-openshift-default-dockercfg-8rh4j\""
Nov 03 00:57:35.661446 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.660460    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"]
Nov 03 00:57:35.757526 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.754007    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.774535    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.774628    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.774669    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.774742    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.774793    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzjm\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.774901    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.775010    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.780457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.775047    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.829546 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.823648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.876668    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.876783    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.876823    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.877032    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.877078    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.877139    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.877173    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.877238    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.877288    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzjm\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.878132    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.878841    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.879103    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.880154    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.883093 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.881235    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.913532 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.913304    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzjm\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.923457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.917838    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.929072 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.928995    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:35.950901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.950864    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:35.952670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.952643    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy\") pod \"e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") " pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:36.011457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:35.988890    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:36.434585 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:57:36.434555    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7d5e68_ac10_426f_8a0a_f5b5d791f875.slice/crio-0ae85f22ce3f6704aab0b3ea3eec3b811ec58ff03462ab0a521a56b7192254b9 WatchSource:0}: Error finding container 0ae85f22ce3f6704aab0b3ea3eec3b811ec58ff03462ab0a521a56b7192254b9: Status 404 returned error can't find the container with id 0ae85f22ce3f6704aab0b3ea3eec3b811ec58ff03462ab0a521a56b7192254b9
Nov 03 00:57:36.437958 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:36.437921    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:36.438060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:36.438000    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:36.438060 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:36.438035    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:57:36.440126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:36.440099    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"]
Nov 03 00:57:36.512143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:36.512110    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" event={"ID":"ee7d5e68-ac10-426f-8a0a-f5b5d791f875","Type":"ContainerStarted","Data":"0ae85f22ce3f6704aab0b3ea3eec3b811ec58ff03462ab0a521a56b7192254b9"}
Nov 03 00:57:36.918049 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:36.916126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:37.014670 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:37.014636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:37.118454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:37.117216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:37.537155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:37.537110    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" event={"ID":"ee7d5e68-ac10-426f-8a0a-f5b5d791f875","Type":"ContainerStarted","Data":"27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"}
Nov 03 00:57:37.611903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:37.611835    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podStartSLOduration=2.611814119 podStartE2EDuration="2.611814119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:37.595732473 +0000 UTC m=+3139.917269273" watchObservedRunningTime="2026-11-03 00:57:37.611814119 +0000 UTC m=+3139.933350921"
Nov 03 00:57:38.024903 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:38.024821    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:38.104798 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:38.104762    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:38.589581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:38.589396    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:38.590590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:38.590424    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:57:38.842821 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:38.842735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:39.303542 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:39.303507    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"]
Nov 03 00:57:39.304036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:39.303763    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" containerID="cri-o://1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105" gracePeriod=30
Nov 03 00:57:39.922114 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:39.922076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:40.043575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:40.043542    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:40.157498 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:40.157449    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:40.261954 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:40.261865    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:40.770178 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:40.770137    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:41.590344 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:41.590300    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:41.723789 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:41.723746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:42.918238 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:42.918202    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:43.335530 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.334687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:43.437465 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.437412    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:43.517267 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.517223    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"]
Nov 03 00:57:43.523475 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.523416    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.539418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.539387    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"]
Nov 03 00:57:43.572447 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.571943    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8ed55a\""
Nov 03 00:57:43.587188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.587022    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8ed55a-dockercfg-brz42\""
Nov 03 00:57:43.665459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.641723    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.665459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.641962    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbpcm\" (UniqueName: \"kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm\") pod \"payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.789858 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.744593    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.790448 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.790392    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbpcm\" (UniqueName: \"kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm\") pod \"payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.799828 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.755750    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.875351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.875212    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbpcm\" (UniqueName: \"kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm\") pod \"payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:43.905145 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.905103    4550 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:57:43.905569 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:43.905541    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:57:44.143131 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.143039    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:44.452797 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.452773    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"]
Nov 03 00:57:44.620084 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.620056    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:57:44.711642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711532    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.711837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711646    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.711837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711728    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.711837 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711812    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.712033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711866    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.712033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711896    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.712033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711934    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.712033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.711978    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.712033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.712010    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:44.712322 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.712062    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data\") pod \"00091f86-a803-44e7-9aed-783ad4016fec\" (UID: \"00091f86-a803-44e7-9aed-783ad4016fec\") "
Nov 03 00:57:44.712918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.712781    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:44.712918 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.712856    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.712964    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.713186    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.713375    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.713399    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.713414    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00091f86-a803-44e7-9aed-783ad4016fec-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.713468    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.713610 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.713482    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.714722 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.714699    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "kube-api-access-rd56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:44.715016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.714980    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:57:44.715016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.714981    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:44.715193 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.715169    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token" pod "00091f86-a803-44e7-9aed-783ad4016fec" (UID: "00091f86-a803-44e7-9aed-783ad4016fec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:44.761651 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.761607    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:44.825454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.818631    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.825454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.818678    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd56h\" (UniqueName: \"kubernetes.io/projected/00091f86-a803-44e7-9aed-783ad4016fec-kube-api-access-rd56h\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.825454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.818696    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00091f86-a803-44e7-9aed-783ad4016fec-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.825454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.818710    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00091f86-a803-44e7-9aed-783ad4016fec-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:57:44.898945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.898910    4550 generic.go:356] "Generic (PLEG): container finished" podID="00091f86-a803-44e7-9aed-783ad4016fec" containerID="1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105" exitCode=0
Nov 03 00:57:44.899140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.898960    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"
Nov 03 00:57:44.899140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.898982    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" event={"ID":"00091f86-a803-44e7-9aed-783ad4016fec","Type":"ContainerDied","Data":"1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105"}
Nov 03 00:57:44.899140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.899046    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9" event={"ID":"00091f86-a803-44e7-9aed-783ad4016fec","Type":"ContainerDied","Data":"e6f00adc31132fc5cb9a6b0c69dd2e7f010cfd9b4114d94eb1180018a942dea3"}
Nov 03 00:57:44.899140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.899073    4550 scope.go:122] "RemoveContainer" containerID="1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105"
Nov 03 00:57:44.901260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.901124    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" event={"ID":"6a97e033-437f-4794-aad7-a9454d4e23a4","Type":"ContainerStarted","Data":"2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84"}
Nov 03 00:57:44.901260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.901171    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" event={"ID":"6a97e033-437f-4794-aad7-a9454d4e23a4","Type":"ContainerStarted","Data":"ca229585579623b9066219184bc6ef905fc54da1dfd115ad14a92ae7fc64509d"}
Nov 03 00:57:44.901260 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.901229    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:44.911456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.911417    4550 scope.go:122] "RemoveContainer" containerID="1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105"
Nov 03 00:57:44.911732 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:57:44.911712    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105\": container with ID starting with 1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105 not found: ID does not exist" containerID="1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105"
Nov 03 00:57:44.911805 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.911744    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105"} err="failed to get container status \"1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105\": rpc error: code = NotFound desc = could not find container \"1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105\": container with ID starting with 1a2d9625a6f0ff3b2fd3e8805a5065b2a7cef7d0a068a960afb14caf1816a105 not found: ID does not exist"
Nov 03 00:57:44.959455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.956812    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" podStartSLOduration=1.956794521 podStartE2EDuration="1.956794521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:44.955655509 +0000 UTC m=+3147.277192312" watchObservedRunningTime="2026-11-03 00:57:44.956794521 +0000 UTC m=+3147.278331323"
Nov 03 00:57:44.997994 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:44.997890    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"]
Nov 03 00:57:45.002389 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:45.002363    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-6lvw9"]
Nov 03 00:57:45.811473 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:45.811419    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00091f86-a803-44e7-9aed-783ad4016fec" path="/var/lib/kubelet/pods/00091f86-a803-44e7-9aed-783ad4016fec/volumes"
Nov 03 00:57:46.436534 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:46.436498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:48.257137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:48.257091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:49.890453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:49.881034    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:51.652776 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:51.652744    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:53.612058 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:53.611997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:54.725305 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:57:54.725264    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:57:54.886933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:54.886899    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"]
Nov 03 00:57:54.887155 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:54.887132    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy" containerID="cri-o://6c99d79172f830c5db450aa4bf5ac00958afef0731f04721f5eb6009681c6dcc" gracePeriod=30
Nov 03 00:57:54.903146 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:54.903106    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"]
Nov 03 00:57:54.903421 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:54.903374    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy" containerID="cri-o://3818e4827cb855145d117c946b4f82041cc7c8fc3606d44d311802784b812f3f" gracePeriod=30
Nov 03 00:57:55.405780 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:55.405747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:55.914846 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:55.914814    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 00:57:57.094719 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:57.094674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:57:58.735079 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:57:58.735026    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:00.126464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.126042    4550 generic.go:356] "Generic (PLEG): container finished" podID="cd14d222-9095-4974-8826-421888f040cf" containerID="6c99d79172f830c5db450aa4bf5ac00958afef0731f04721f5eb6009681c6dcc" exitCode=0
Nov 03 00:58:00.126464 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.126300    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" event={"ID":"cd14d222-9095-4974-8826-421888f040cf","Type":"ContainerDied","Data":"6c99d79172f830c5db450aa4bf5ac00958afef0731f04721f5eb6009681c6dcc"}
Nov 03 00:58:00.128728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.128697    4550 generic.go:356] "Generic (PLEG): container finished" podID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerID="3818e4827cb855145d117c946b4f82041cc7c8fc3606d44d311802784b812f3f" exitCode=0
Nov 03 00:58:00.128868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.128851    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" event={"ID":"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8","Type":"ContainerDied","Data":"3818e4827cb855145d117c946b4f82041cc7c8fc3606d44d311802784b812f3f"}
Nov 03 00:58:00.231112 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.231077    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:58:00.278952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.278883    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:58:00.360893 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.360852    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.360937    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.360978    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361037    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361082 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361077    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361125    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361198    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361226    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361228    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.361331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361292    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361331 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361330    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361365    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361404    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361449    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361470    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361577    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361603    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.361672 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361657    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert\") pod \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\" (UID: \"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8\") "
Nov 03 00:58:00.362016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361687    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.362016 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.361717    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5\") pod \"cd14d222-9095-4974-8826-421888f040cf\" (UID: \"cd14d222-9095-4974-8826-421888f040cf\") "
Nov 03 00:58:00.363059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.362313    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:58:00.363059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.362516    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.363059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.362538    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.363059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.362553    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd14d222-9095-4974-8826-421888f040cf-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.363059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.362850    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.363407 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.363377    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.369449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.367911    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "kube-api-access-jhjb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:58:00.369449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.367950    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:58:00.369449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.368189    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.369449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.368233    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:58:00.369449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.368823    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:58:00.369449 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.369087    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.370644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.369488    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:58:00.370644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.370564    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:58:00.371494 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.371472    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.379644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.379614    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket" pod "cd14d222-9095-4974-8826-421888f040cf" (UID: "cd14d222-9095-4974-8826-421888f040cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.380642 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.380620    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.383271 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.383245    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "kube-api-access-bfgcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:58:00.384677 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.384655    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy" pod "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" (UID: "ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:58:00.397316 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.397205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.494902    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.494940    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgcd\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-kube-api-access-bfgcd\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.494955    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.494966    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.494978    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.494989    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495004    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495016    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd14d222-9095-4974-8826-421888f040cf-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495026    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495037    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495048    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495060    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495074    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495085    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd14d222-9095-4974-8826-421888f040cf-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:00.499539 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:00.495096    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhjb5\" (UniqueName: \"kubernetes.io/projected/cd14d222-9095-4974-8826-421888f040cf-kube-api-access-jhjb5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:58:01.138580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.138542    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"
Nov 03 00:58:01.138580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.138553    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg" event={"ID":"cd14d222-9095-4974-8826-421888f040cf","Type":"ContainerDied","Data":"0193aa8acf176dd493b4e43c11ffaa8f0bddbe5eb768bd1a802c74246e483cfb"}
Nov 03 00:58:01.139140 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.138634    4550 scope.go:122] "RemoveContainer" containerID="6c99d79172f830c5db450aa4bf5ac00958afef0731f04721f5eb6009681c6dcc"
Nov 03 00:58:01.140452 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.140421    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"
Nov 03 00:58:01.140554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.140457    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926" event={"ID":"ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8","Type":"ContainerDied","Data":"9f15453b9fac85be1f2117e030eb325744d3fc72f6fc7577bc81453dfcf35bed"}
Nov 03 00:58:01.152291 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.152273    4550 scope.go:122] "RemoveContainer" containerID="3818e4827cb855145d117c946b4f82041cc7c8fc3606d44d311802784b812f3f"
Nov 03 00:58:01.198198 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.198108    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"]
Nov 03 00:58:01.209725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.209689    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-6q926"]
Nov 03 00:58:01.234471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.234409    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"]
Nov 03 00:58:01.239862 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.239836    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-r2qqg"]
Nov 03 00:58:01.811343 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.811307    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd14d222-9095-4974-8826-421888f040cf" path="/var/lib/kubelet/pods/cd14d222-9095-4974-8826-421888f040cf/volumes"
Nov 03 00:58:01.811888 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:01.811873    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" path="/var/lib/kubelet/pods/ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8/volumes"
Nov 03 00:58:02.080338 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:02.080246    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:03.727869 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:03.727828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:06.742075 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:06.742045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:07.304967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:07.304929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:08.936189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:08.936135    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:10.633375 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:10.633336    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:12.442459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:12.442196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:13.881077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:13.881039    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:14.235491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:14.235455    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:15.931345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:15.931306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:17.533006 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:17.532969    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:19.199073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:19.199030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:19.294409 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:19.294356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:20.667694 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:20.667660    4550 scope.go:122] "RemoveContainer" containerID="1e6d6192cc7505a3fa42fb8cc6ce602058b5624e80b2f1286902a0f76ecaf1a7"
Nov 03 00:58:20.688059 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:20.688026    4550 scope.go:122] "RemoveContainer" containerID="5671da8470cdbf82ec09d60540772de4148a757c26a768823a9230ab0c263309"
Nov 03 00:58:20.700171 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:20.700147    4550 scope.go:122] "RemoveContainer" containerID="ac67cf724d3112a376f511a64c8d42525e180e870e6a574a64af91857895e987"
Nov 03 00:58:20.952366 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:20.952273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:22.642758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:22.642707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:24.315933 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:24.315894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:24.726050 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:58:24.726008    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:58:25.960892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:25.960855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:27.634905 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:27.634868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:29.221718 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:29.221683    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:30.860927 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:30.860890    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:32.541153 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:32.541112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:34.354253 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:34.354158    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:34.442032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:34.441990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:34.541594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:34.541563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:36.300345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:36.300247    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:36.423454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:36.406885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:36.508033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:36.507951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:36.789057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:36.788863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:38.237033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:38.236989    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:39.723200 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:39.723164    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:40.021177 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:40.021049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:41.708998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:41.708954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:43.344868 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:43.344828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:44.154099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.152188    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": read tcp 10.133.0.2:49498->10.133.0.227:15021: read: connection reset by peer" start-of-body=
Nov 03 00:58:44.154099 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.152269    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": read tcp 10.133.0.2:49498->10.133.0.227:15021: read: connection reset by peer"
Nov 03 00:58:44.648218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.648127    4550 generic.go:356] "Generic (PLEG): container finished" podID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerID="0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a" exitCode=0
Nov 03 00:58:44.648218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.648202    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" event={"ID":"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0","Type":"ContainerDied","Data":"0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"}
Nov 03 00:58:44.648778 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.648761    4550 scope.go:122] "RemoveContainer" containerID="0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"
Nov 03 00:58:44.651073 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.651043    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:58:44.651158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.651114    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:58:44.651158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:44.651146    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:58:45.035424 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:45.035387    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:45.413351 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:45.413177    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:45.715521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:45.714967    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" event={"ID":"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0","Type":"ContainerStarted","Data":"e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce"}
Nov 03 00:58:46.237235 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:46.237198    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:58:46.237517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:46.237494    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:58:46.237580 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:46.237556    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:58:46.788650 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:46.788593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:47.244887 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:47.244846    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:58:47.245225 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:47.245186    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:58:48.236964 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:48.236928    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:58:48.237379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:48.236992    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:58:48.530643 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:48.530544    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:49.237074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:49.237037    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:58:49.237536 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:49.237098    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:58:50.194557 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:50.194520    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:50.237104 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:50.237067    4550 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:58:50.237553 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:50.237119    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused"
Nov 03 00:58:51.321372 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:51.321342    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:58:51.322796 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:51.322720    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:58:51.323877 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:51.323854    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 00:58:51.956676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:51.956639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:53.715478 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:53.715412    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:54.726832 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:58:54.726795    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:58:55.346047 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:55.346014    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:57.011998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:57.011954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:58:58.625057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:58:58.625020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:00.324180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:00.324148    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:01.950216 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:01.950180    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:02.944306 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:02.944273    4550 generic.go:356] "Generic (PLEG): container finished" podID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerID="6761bec57d1b47678c389457d3985f2dbc8a8aa10ea0865dab68aaf0bd194d97" exitCode=0
Nov 03 00:59:02.944568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:02.944346    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" event={"ID":"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a","Type":"ContainerDied","Data":"6761bec57d1b47678c389457d3985f2dbc8a8aa10ea0865dab68aaf0bd194d97"}
Nov 03 00:59:02.944853 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:02.944834    4550 scope.go:122] "RemoveContainer" containerID="6761bec57d1b47678c389457d3985f2dbc8a8aa10ea0865dab68aaf0bd194d97"
Nov 03 00:59:02.946930 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:02.946906    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:02.947018 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:02.946973    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:02.947068 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:02.947025    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:03.673943 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:03.673907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:03.954633 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:03.954550    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" event={"ID":"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a","Type":"ContainerStarted","Data":"7686334f0bb0660b40d9bbd046a682abed2b68217001edde795b655607d3cb50"}
Nov 03 00:59:04.441514 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:04.441479    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:59:04.442967 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:04.442938    4550 patch_prober.go:29] interesting pod/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 00:59:04.443115 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:04.442994    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 00:59:05.441996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:05.441958    4550 patch_prober.go:29] interesting pod/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 00:59:05.442466 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:05.442028    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 00:59:05.448156 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:05.448127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:06.030166 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:06.030122    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"]
Nov 03 00:59:06.042422 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:06.042388    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0050a8e-wjwfv"]
Nov 03 00:59:06.442033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:06.441996    4550 patch_prober.go:29] interesting pod/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 00:59:06.442575 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:06.442068    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 00:59:07.104347 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:07.104312    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:07.442033 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:07.441997    4550 patch_prober.go:29] interesting pod/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 00:59:07.442232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:07.442060    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 00:59:07.813119 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:07.813045    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d273ce19-d14e-452e-a2d2-831ce855f063" path="/var/lib/kubelet/pods/d273ce19-d14e-452e-a2d2-831ce855f063/volumes"
Nov 03 00:59:08.227963 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:08.227924    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:59:08.442503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:08.442424    4550 patch_prober.go:29] interesting pod/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 00:59:08.442916 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:08.442514    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 00:59:08.786419 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:08.786384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:09.442032 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:09.441995    4550 patch_prober.go:29] interesting pod/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused" start-of-body=
Nov 03 00:59:09.442240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:09.442083    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.237:15021/healthz/ready\": dial tcp 10.133.0.237:15021: connect: connection refused"
Nov 03 00:59:10.447283 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:10.447248    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:59:10.448535 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:10.448510    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 00:59:10.522912 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:10.522880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:12.197471 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:12.197416    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:13.912996 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:13.912958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:14.024558 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:14.024411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:14.136139 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:14.136102    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:15.527623 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:15.527591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:17.202640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:17.202600    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:18.033945 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:18.033901    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"]
Nov 03 00:59:18.085244 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:18.075294    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-924984da-l57v2"]
Nov 03 00:59:18.965237 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:18.965145    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:19.853147 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:19.853103    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e47dee8-72a2-4214-afa4-94dba004e66f" path="/var/lib/kubelet/pods/7e47dee8-72a2-4214-afa4-94dba004e66f/volumes"
Nov 03 00:59:21.404350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:21.404306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:22.325511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:22.325475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:23.236669 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:23.236637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:24.729034 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:59:24.728991    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:59:25.050125 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.050030    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:25.437476 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.437420    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"]
Nov 03 00:59:25.438600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438574    4550 cpu_manager.go:414] "containerMap: removing container" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy"
Nov 03 00:59:25.438714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438603    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy"
Nov 03 00:59:25.438714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438629    4550 cpu_manager.go:414] "containerMap: removing container" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy"
Nov 03 00:59:25.438714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438638    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy"
Nov 03 00:59:25.438714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438708    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy"
Nov 03 00:59:25.438714 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438715    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy"
Nov 03 00:59:25.439008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438883    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9f4edf-fbac-4a91-b9a1-0dc3b7caa9b8" containerName="istio-proxy"
Nov 03 00:59:25.439008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438904    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd14d222-9095-4974-8826-421888f040cf" containerName="istio-proxy"
Nov 03 00:59:25.439008 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.438915    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="00091f86-a803-44e7-9aed-783ad4016fec" containerName="istio-proxy"
Nov 03 00:59:25.443505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.443417    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.468057 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.468025    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"]
Nov 03 00:59:25.468301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.468079    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49d7e5\"/\"openshift-service-ca.crt\""
Nov 03 00:59:25.468418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.468079    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-49d7e5\"/\"kube-root-ca.crt\""
Nov 03 00:59:25.476500 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.476473    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49d7e5\"/\"default-dockercfg-c2zrn\""
Nov 03 00:59:25.476656 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.476503    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-49d7e5\"/\"test-model-49d7e5-kserve-self-signed-certs\""
Nov 03 00:59:25.481739 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.481711    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:25.565827 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.565790    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.566063 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.566043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.566180 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.566164    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.566256 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.566206    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhlh\" (UniqueName: \"kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.566350 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.566327    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.566458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.566391    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.674724    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.674832    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.674871    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhlh\" (UniqueName: \"kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.674952    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.675010    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.675178    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.676588 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.676334    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.705503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.696188    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.705503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.699770    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.727453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.717455    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.727453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.717763    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.727453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.718418    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhlh\" (UniqueName: \"kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh\") pod \"test-model-49d7e5-kserve-5bd87ddd55-g7hgz\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") " pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:25.770457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:25.769974    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:26.190149 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:59:26.190112    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode472ade3_5a80_4839_8704_e627869fae14.slice/crio-806c744d65001a81f6b53ec4b10829f4a1291cb3b155ad99d31d6e9128be480d WatchSource:0}: Error finding container 806c744d65001a81f6b53ec4b10829f4a1291cb3b155ad99d31d6e9128be480d: Status 404 returned error can't find the container with id 806c744d65001a81f6b53ec4b10829f4a1291cb3b155ad99d31d6e9128be480d
Nov 03 00:59:26.194742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:26.194710    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"]
Nov 03 00:59:26.240582 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:26.240542    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz" event={"ID":"e472ade3-5a80-4839-8704-e627869fae14","Type":"ContainerStarted","Data":"806c744d65001a81f6b53ec4b10829f4a1291cb3b155ad99d31d6e9128be480d"}
Nov 03 00:59:26.915348 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:26.915313    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:27.250554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:27.250460    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz" event={"ID":"e472ade3-5a80-4839-8704-e627869fae14","Type":"ContainerStarted","Data":"55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae"}
Nov 03 00:59:27.291274 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:27.291212    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz" podStartSLOduration=2.291199623 podStartE2EDuration="2.291199623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:27.290331798 +0000 UTC m=+3249.611868601" watchObservedRunningTime="2026-11-03 00:59:27.291199623 +0000 UTC m=+3249.612736422"
Nov 03 00:59:27.770496 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:27.770427    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:28.677135 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:28.677086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:30.283379 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:30.283341    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:32.129527 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:32.129491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:34.022595 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:34.022556    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:35.770952 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:35.770909    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:35.783803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:35.783772    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:35.806484 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:35.806453    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:36.345123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:36.345093    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 00:59:37.686014 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:37.683658    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:37.780218 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:37.780184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:38.148568 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.148462    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:38.559459 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.556188    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"]
Nov 03 00:59:38.597777 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.597744    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.609573 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.606058    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:38.619170 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.619136    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"]
Nov 03 00:59:38.620213 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620108    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.620371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620254    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.620371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620307    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.620371 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620348    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.621728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620480    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvrj\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.621728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620546    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.621728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620631    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.621728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620696    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.621728 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.620737    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.627123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.627096    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-537652-openshift-default-dockercfg-kq8vr\""
Nov 03 00:59:38.722492 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.722444    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.730558    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.730630    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.730743    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvrj\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.730818    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.730925    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.731005    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.731053    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.750456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.731107    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.820607    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.820840    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.820929    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.821107    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.821217    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.821784    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:38.847547 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:38.822644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:39.139521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.136226    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:39.139521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.137207    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvrj\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:39.139521 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.138100    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo\") pod \"e2e-inf-b-537652-openshift-default-746478548f-zvwjz\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") " pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:39.236743 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.236704    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:39.547920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.547885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:39.599612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.599054    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"]
Nov 03 00:59:39.634067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.627352    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:39.634067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.627484    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:39.634067 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:39.627530    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:40.410576 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:40.410539    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" event={"ID":"faa828ae-d375-4578-9485-71ed468a41e1","Type":"ContainerStarted","Data":"ce701d4ad0fe7a60fd290e10f34560f442e2c3aaa49638b3940f937ea0af183c"}
Nov 03 00:59:40.411074 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:40.410591    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" event={"ID":"faa828ae-d375-4578-9485-71ed468a41e1","Type":"ContainerStarted","Data":"3a2951c014fc183f0a2404297a8d917fd1b7496f11a9cf839d0bcddb994e9c97"}
Nov 03 00:59:40.455304 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:40.455246    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podStartSLOduration=2.455231782 podStartE2EDuration="2.455231782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:40.452017116 +0000 UTC m=+3262.773553917" watchObservedRunningTime="2026-11-03 00:59:40.455231782 +0000 UTC m=+3262.776768655"
Nov 03 00:59:41.237639 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:41.237596    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:41.244418 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:41.244385    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:41.254161 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:41.254125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:41.420392 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:41.420360    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:41.421638 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:41.421615    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 00:59:42.840979 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:42.840947    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:42.996467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:42.996409    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:43.422906 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.422862    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:43.542969 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.542914    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:43.783503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.783465    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"]
Nov 03 00:59:43.791329 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.791299    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:43.845036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.844985    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-70c7d6-openshift-default-dockercfg-d68wt\""
Nov 03 00:59:43.851819 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.851713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:43.861158 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:43.861108    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"]
Nov 03 00:59:44.137884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.137783    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.137884 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.137838    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.137897    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.137921    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.137963    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.138101    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdv7t\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138194 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.138150    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.138260    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.138571 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.138297    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.240286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.240241    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.241168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.241136    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.248137 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.248097    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.248844 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.248815    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.249301 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.249283    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.249600 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.249581    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdv7t\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.249755 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.249741    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.249956 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.249942    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.250077 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.250061    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.250317 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.249223    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.250537 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.248753    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.250678 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.250648    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.251469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.251449    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.251874 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.251856    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.252934 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.252907    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.257097 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.257054    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.272307 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.272271    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdv7t\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.279939 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.279882    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token\") pod \"e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") " pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.313399 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.313359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:44.504386 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.504332    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:44.841112 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:59:44.841073    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d304809_13c7_4fff_8287_b781199a7e3d.slice/crio-619ab0fd8c284010821116f6b5a20eabc5d44c3f6bbfc3fdf1c82d8a7c06c5c1 WatchSource:0}: Error finding container 619ab0fd8c284010821116f6b5a20eabc5d44c3f6bbfc3fdf1c82d8a7c06c5c1: Status 404 returned error can't find the container with id 619ab0fd8c284010821116f6b5a20eabc5d44c3f6bbfc3fdf1c82d8a7c06c5c1
Nov 03 00:59:44.842965 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.842899    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"]
Nov 03 00:59:44.844583 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.844549    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:44.844674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.844625    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:44.844674 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:44.844657    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:45.167782 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.167735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:45.282630 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.282582    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:45.287255 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.287218    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"]
Nov 03 00:59:45.293308 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.293017    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.316240 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.316105    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"]
Nov 03 00:59:45.316696 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.316671    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-537652\""
Nov 03 00:59:45.390543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.390505    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.391029 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.390998    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65n5\" (UniqueName: \"kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5\") pod \"payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.507505 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.507325    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65n5\" (UniqueName: \"kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5\") pod \"payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.507704 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.507563    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.528458 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.508630    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.545455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.539945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:45.605457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.599298    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65n5\" (UniqueName: \"kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5\") pod \"payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.633455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.610083    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" event={"ID":"4d304809-13c7-4fff-8287-b781199a7e3d","Type":"ContainerStarted","Data":"713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"}
Nov 03 00:59:45.633455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.610151    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" event={"ID":"4d304809-13c7-4fff-8287-b781199a7e3d","Type":"ContainerStarted","Data":"619ab0fd8c284010821116f6b5a20eabc5d44c3f6bbfc3fdf1c82d8a7c06c5c1"}
Nov 03 00:59:45.633455 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.632038    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"]
Nov 03 00:59:45.658454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.641712    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:45.682456 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.677945    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podStartSLOduration=2.677923459 podStartE2EDuration="2.677923459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:45.658592062 +0000 UTC m=+3267.980128863" watchObservedRunningTime="2026-11-03 00:59:45.677923459 +0000 UTC m=+3267.999460261"
Nov 03 00:59:45.825188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.775636    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"]
Nov 03 00:59:45.825188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.775771    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:45.825188 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.792636    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-537652-dockercfg-czvdh\""
Nov 03 00:59:45.828010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.827422    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:45.828010 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.827611    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrwkv\" (UniqueName: \"kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv\") pod \"payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:45.941136 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.930148    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:45.941136 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.930237    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrwkv\" (UniqueName: \"kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv\") pod \"payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:45.941136 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.939599    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:45.987036 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:45.986995    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrwkv\" (UniqueName: \"kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv\") pod \"payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") " pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:46.167944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.164328    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:46.300124 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:59:46.300050    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2d13d0_101e_405d_8ca5_87e709269cfc.slice/crio-d526d674764d88c8c813846a168b68483ca87c87f530dc1b585568f053a822d7 WatchSource:0}: Error finding container d526d674764d88c8c813846a168b68483ca87c87f530dc1b585568f053a822d7: Status 404 returned error can't find the container with id d526d674764d88c8c813846a168b68483ca87c87f530dc1b585568f053a822d7
Nov 03 00:59:46.301231 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.301163    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"]
Nov 03 00:59:46.500355 ip-10-0-133-253 kubenswrapper[4550]: W1103 00:59:46.500317    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfda3998_0005_44bb_9dad_f09c1ae69178.slice/crio-9c782bc00d64456f4e116d6e950f6df556cece327ae36c33347de8d3a618d95c WatchSource:0}: Error finding container 9c782bc00d64456f4e116d6e950f6df556cece327ae36c33347de8d3a618d95c: Status 404 returned error can't find the container with id 9c782bc00d64456f4e116d6e950f6df556cece327ae36c33347de8d3a618d95c
Nov 03 00:59:46.504109 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.504084    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"]
Nov 03 00:59:46.506734 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.504622    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:46.524182 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.519724    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:46.623581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.623524    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" event={"ID":"ec2d13d0-101e-405d-8ca5-87e709269cfc","Type":"ContainerStarted","Data":"45699819513af5f41c54eaccb86837d13b4b2b3c97f9b92a4037f17ebca4a0ae"}
Nov 03 00:59:46.623725 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.623598    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" event={"ID":"ec2d13d0-101e-405d-8ca5-87e709269cfc","Type":"ContainerStarted","Data":"d526d674764d88c8c813846a168b68483ca87c87f530dc1b585568f053a822d7"}
Nov 03 00:59:46.624232 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.624212    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:46.626015 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.625987    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" event={"ID":"dfda3998-0005-44bb-9dad-f09c1ae69178","Type":"ContainerStarted","Data":"9c782bc00d64456f4e116d6e950f6df556cece327ae36c33347de8d3a618d95c"}
Nov 03 00:59:46.626508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.626489    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:46.628070 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.628050    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 00:59:46.665167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:46.665103    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" podStartSLOduration=1.665084095 podStartE2EDuration="1.665084095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:46.663714281 +0000 UTC m=+3268.985251083" watchObservedRunningTime="2026-11-03 00:59:46.665084095 +0000 UTC m=+3268.986620899"
Nov 03 00:59:47.579612 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:47.579517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:47.667420 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:47.667354    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" event={"ID":"dfda3998-0005-44bb-9dad-f09c1ae69178","Type":"ContainerStarted","Data":"1572511217e237c2e9d04e1bc8811169a31d33fd7663dd100c18c9cf16f7bb6e"}
Nov 03 00:59:47.668467 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:47.668426    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 00:59:47.714820 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:47.714752    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" podStartSLOduration=2.71473702 podStartE2EDuration="2.71473702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:47.712556767 +0000 UTC m=+3270.034093569" watchObservedRunningTime="2026-11-03 00:59:47.71473702 +0000 UTC m=+3270.036273821"
Nov 03 00:59:48.148944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:48.148907    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:48.260450 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:48.256670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:48.495453 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:48.494611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:49.234554 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.234515    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"]
Nov 03 00:59:49.239483 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.239458    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.262591 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.262554    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"]
Nov 03 00:59:49.270715 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.270683    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 00:59:49.270926 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.270794    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:59:49.328590 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.328455    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.328878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.328590    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rln2\" (UniqueName: \"kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.328878 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.328653    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.430609 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.430568    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rln2\" (UniqueName: \"kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.430803 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.430630    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.430892 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.430862    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.431403 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.431374    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.433298 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.433275    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.489364 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.489331    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rln2\" (UniqueName: \"kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2\") pod \"maas-api-revoke-sub-c9c38501-2xcgm\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.557545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.557508    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:49.718543 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.718264    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:49.741962 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.741931    4550 generic.go:356] "Generic (PLEG): container finished" podID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerID="27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001" exitCode=0
Nov 03 00:59:49.742209 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.742180    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" event={"ID":"ee7d5e68-ac10-426f-8a0a-f5b5d791f875","Type":"ContainerDied","Data":"27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"}
Nov 03 00:59:49.742767 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.742749    4550 scope.go:122] "RemoveContainer" containerID="27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"
Nov 03 00:59:49.745503 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.745470    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:49.745604 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.745563    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:49.745649 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.745611    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 00:59:49.880920 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:49.880884    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"]
Nov 03 00:59:50.389511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.388361    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:50.484594 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.484560    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:50.505204 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.505168    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"]
Nov 03 00:59:50.509899 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.509873    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.525817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.525782    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-70c7d6\""
Nov 03 00:59:50.525817 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.525803    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-70c7d6-dockercfg-rcc47\""
Nov 03 00:59:50.531901 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.531872    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"]
Nov 03 00:59:50.549451 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.549388    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.549665 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.549461    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9vg\" (UniqueName: \"kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg\") pod \"payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.573195 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.573000    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"]
Nov 03 00:59:50.582454 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.579242    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:50.612123 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.612087    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"]
Nov 03 00:59:50.679790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.677948    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.679790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.678028    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9vg\" (UniqueName: \"kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg\") pod \"payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.679790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.678199    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:50.679790 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.678399    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqtp\" (UniqueName: \"kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp\") pod \"payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:50.695486 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.680262    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.787457 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.768159    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9vg\" (UniqueName: \"kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg\") pod \"payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.855460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.845982    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:50.855460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.849745    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:50.855460 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.849870    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqtp\" (UniqueName: \"kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp\") pod \"payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:50.907657 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.907622    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" event={"ID":"ee7d5e68-ac10-426f-8a0a-f5b5d791f875","Type":"ContainerStarted","Data":"3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2"}
Nov 03 00:59:50.917189 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.917143    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm" event={"ID":"843a9551-8b8c-420e-aa9c-166fad1f67ce","Type":"ContainerStarted","Data":"1202e1c4e8384f56a2b0989a39c65ce160528ee2e3acc28d7663d1f086b9a237"}
Nov 03 00:59:50.918559 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.917478    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm" event={"ID":"843a9551-8b8c-420e-aa9c-166fad1f67ce","Type":"ContainerStarted","Data":"033ad480b106d23a93dddbc39c707362eb31307a068ff09863d03dc124f68d18"}
Nov 03 00:59:50.928297 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.928261    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:50.979997 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:50.970176    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqtp\" (UniqueName: \"kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp\") pod \"payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:51.017143 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.017060    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:59:51.018487 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.018456    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:51.018622 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.018511    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:51.023561 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.023491    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm" podStartSLOduration=2.023471469 podStartE2EDuration="2.023471469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:51.016781223 +0000 UTC m=+3273.338318026" watchObservedRunningTime="2026-11-03 00:59:51.023471469 +0000 UTC m=+3273.345008272"
Nov 03 00:59:51.218581 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.218541    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:51.348742 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.347398    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"]
Nov 03 00:59:51.541932 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.541904    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"]
Nov 03 00:59:51.640506 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.640473    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:51.926758 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.926722    4550 generic.go:356] "Generic (PLEG): container finished" podID="843a9551-8b8c-420e-aa9c-166fad1f67ce" containerID="1202e1c4e8384f56a2b0989a39c65ce160528ee2e3acc28d7663d1f086b9a237" exitCode=0
Nov 03 00:59:51.926948 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.926809    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm" event={"ID":"843a9551-8b8c-420e-aa9c-166fad1f67ce","Type":"ContainerDied","Data":"1202e1c4e8384f56a2b0989a39c65ce160528ee2e3acc28d7663d1f086b9a237"}
Nov 03 00:59:51.928666 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.928637    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p" event={"ID":"ec65abdf-e24f-4390-a5fb-15b563cde55d","Type":"ContainerStarted","Data":"3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109"}
Nov 03 00:59:51.928809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.928683    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p" event={"ID":"ec65abdf-e24f-4390-a5fb-15b563cde55d","Type":"ContainerStarted","Data":"ede794e16fcf3abe0f78a5e6cd05b9b44e4a41d197393bd510ff6420e57c14c8"}
Nov 03 00:59:51.928809 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.928736    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 00:59:51.930380 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.930358    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" event={"ID":"50aa2c23-760a-4be5-82b9-bbbd90b4ad64","Type":"ContainerStarted","Data":"7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51"}
Nov 03 00:59:51.930524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.930396    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" event={"ID":"50aa2c23-760a-4be5-82b9-bbbd90b4ad64","Type":"ContainerStarted","Data":"815fbf5172fa0f00658024bd3128de4436b572b4f11f36394f4e8b816cde32b9"}
Nov 03 00:59:51.930524 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:51.930482    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 00:59:52.015881 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:52.015846    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:52.016080 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:52.015906    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:52.019889 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:52.019826    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p" podStartSLOduration=2.019808044 podStartE2EDuration="2.019808044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:52.016666259 +0000 UTC m=+3274.338203081" watchObservedRunningTime="2026-11-03 00:59:52.019808044 +0000 UTC m=+3274.341344847"
Nov 03 00:59:52.077382 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:52.077307    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" podStartSLOduration=2.077286087 podStartE2EDuration="2.077286087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:52.075956414 +0000 UTC m=+3274.397493230" watchObservedRunningTime="2026-11-03 00:59:52.077286087 +0000 UTC m=+3274.398822888"
Nov 03 00:59:53.015998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.015910    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:53.016545 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.016036    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:53.030406 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.030374    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"]
Nov 03 00:59:53.049491 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.047872    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2ff8ac3-7m9tj"]
Nov 03 00:59:53.218998 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.218938    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:53.332818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.332004    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca\") pod \"843a9551-8b8c-420e-aa9c-166fad1f67ce\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") "
Nov 03 00:59:53.332818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.332075    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca\") pod \"843a9551-8b8c-420e-aa9c-166fad1f67ce\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") "
Nov 03 00:59:53.332818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.332138    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2\" (UniqueName: \"kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2\") pod \"843a9551-8b8c-420e-aa9c-166fad1f67ce\" (UID: \"843a9551-8b8c-420e-aa9c-166fad1f67ce\") "
Nov 03 00:59:53.332818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.332404    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca" pod "843a9551-8b8c-420e-aa9c-166fad1f67ce" (UID: "843a9551-8b8c-420e-aa9c-166fad1f67ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:59:53.332818 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.332731    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:59:53.334644 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.334567    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca" pod "843a9551-8b8c-420e-aa9c-166fad1f67ce" (UID: "843a9551-8b8c-420e-aa9c-166fad1f67ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:59:53.335002 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.334969    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2" pod "843a9551-8b8c-420e-aa9c-166fad1f67ce" (UID: "843a9551-8b8c-420e-aa9c-166fad1f67ce"). InnerVolumeSpecName "kube-api-access-5rln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:59:53.343517 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.343489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:53.442224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.441099    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843a9551-8b8c-420e-aa9c-166fad1f67ce-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:59:53.442224 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.441145    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rln2\" (UniqueName: \"kubernetes.io/projected/843a9551-8b8c-420e-aa9c-166fad1f67ce-kube-api-access-5rln2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 00:59:53.592511 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.591542    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:59:53.813684 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.813649    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e3865f-38e4-4eb5-af83-71171c64cda0" path="/var/lib/kubelet/pods/47e3865f-38e4-4eb5-af83-71171c64cda0/volumes"
Nov 03 00:59:53.948944 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.948919    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"
Nov 03 00:59:53.949222 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.948958    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm" event={"ID":"843a9551-8b8c-420e-aa9c-166fad1f67ce","Type":"ContainerDied","Data":"033ad480b106d23a93dddbc39c707362eb31307a068ff09863d03dc124f68d18"}
Nov 03 00:59:53.949286 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:53.949248    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033ad480b106d23a93dddbc39c707362eb31307a068ff09863d03dc124f68d18"
Nov 03 00:59:54.015825 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:54.015786    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:54.016025 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:54.015844    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:54.726209 ip-10-0-133-253 kubenswrapper[4550]: E1103 00:59:54.726172    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 00:59:55.007508 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:55.006705    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:55.015676 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:55.015642    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:55.015909 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:55.015874    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:56.015640 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:56.015595    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:56.016126 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:56.015651    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:56.805469 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:56.803251    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:57.015924 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:57.015885    4550 patch_prober.go:29] interesting pod/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body=
Nov 03 00:59:57.016345 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:57.015970    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused"
Nov 03 00:59:57.670619 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:57.670590    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 00:59:58.021167 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:58.021091    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:59:58.022168 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:58.022145    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 00:59:58.465085 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:58.465052    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 00:59:59.746342 ip-10-0-133-253 kubenswrapper[4550]: I1103 00:59:59.746249    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 01:00:00.115194 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.115091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:00.188206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.188171    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"]
Nov 03 01:00:00.198389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.198359    4550 cpu_manager.go:414] "containerMap: removing container" podUID="843a9551-8b8c-420e-aa9c-166fad1f67ce" containerName="revoke-keys"
Nov 03 01:00:00.198718 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.198699    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843a9551-8b8c-420e-aa9c-166fad1f67ce" containerName="revoke-keys"
Nov 03 01:00:00.199213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.199196    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="843a9551-8b8c-420e-aa9c-166fad1f67ce" containerName="revoke-keys"
Nov 03 01:00:00.211400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.211367    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"]
Nov 03 01:00:00.211731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.211579    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.234250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.229469    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"]
Nov 03 01:00:00.235039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.234879    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.248534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.248473    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.253832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.253773    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:00:00.276157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.276115    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"]
Nov 03 01:00:00.285883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.285849    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"]
Nov 03 01:00:00.290578 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.290548    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.301537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.301285    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"]
Nov 03 01:00:00.301792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.301765    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.308408 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.308369    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"]
Nov 03 01:00:00.324515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.324482    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"]
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.388482    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"]
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.390196    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.408981    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt5mb\" (UniqueName: \"kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409052    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7rh\" (UniqueName: \"kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh\") pod \"maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409106    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409173    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4mr\" (UniqueName: \"kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409264    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409291    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvws7\" (UniqueName: \"kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7\") pod \"maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409346    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.410506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.409377    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkzz\" (UniqueName: \"kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.413828 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.413788    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.414152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.414126    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.437182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.437144    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"]
Nov 03 01:00:00.441278 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.441245    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.450346 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.450306    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"]
Nov 03 01:00:00.452146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.452112    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.464563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.464533    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"]
Nov 03 01:00:00.466521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.464899    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.474603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.474582    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"]
Nov 03 01:00:00.474793 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.474779    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"]
Nov 03 01:00:00.475043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.475027    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.481514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.481484    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"]
Nov 03 01:00:00.492999 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.491847    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"]
Nov 03 01:00:00.507405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.506187    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.514625 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.513350    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"]
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517313    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517382    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvws7\" (UniqueName: \"kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7\") pod \"maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517488    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517561    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517610    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkzz\" (UniqueName: \"kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517670    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.517801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517777    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.518267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.517811    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scppx\" (UniqueName: \"kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.518267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.518025    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt5mb\" (UniqueName: \"kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.518267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.518086    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7rh\" (UniqueName: \"kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh\") pod \"maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.518267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.518142    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.520784 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.520634    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.521533 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.521504    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.521640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.521615    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4mr\" (UniqueName: \"kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.522989 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.522968    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.523173 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.523147    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.523824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.523789    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.528310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.528292    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"]
Nov 03 01:00:00.533412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.528522    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.538748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.538727    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.560266 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.560235    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"]
Nov 03 01:00:00.584076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.565406    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7rh\" (UniqueName: \"kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh\") pod \"maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.587522 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.584933    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4mr\" (UniqueName: \"kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.589054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.588304    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt5mb\" (UniqueName: \"kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb\") pod \"maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.589895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.589871    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvws7\" (UniqueName: \"kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7\") pod \"maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.594422 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.594272    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"]
Nov 03 01:00:00.594637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.594396    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.603209 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.603188    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"]
Nov 03 01:00:00.603474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.603421    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"]
Nov 03 01:00:00.603798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.603325    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.604609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.604582    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkzz\" (UniqueName: \"kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz\") pod \"maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.609892 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.609866    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"]
Nov 03 01:00:00.610170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.610084    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.615236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.613752    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:00.617334 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.617311    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"]
Nov 03 01:00:00.618537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.618276    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:00.631613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.631369    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:00.649759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.649505    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:00.668402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.650365    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxjzx\" (UniqueName: \"kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx\") pod \"maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.692698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.670045    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.698047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.697073    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.734950    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735060    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw78s\" (UniqueName: \"kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735135    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735162    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735221    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qbvp\" (UniqueName: \"kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735252    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz6vw\" (UniqueName: \"kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735311    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9h6\" (UniqueName: \"kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6\") pod \"maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735407    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735452    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735480    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735521    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scppx\" (UniqueName: \"kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735568    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735596    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-7bhb4\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735748    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn4bn\" (UniqueName: \"kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn\") pod \"maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735822    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxl4x\" (UniqueName: \"kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x\") pod \"maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735859    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735928    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xp5z\" (UniqueName: \"kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.735975    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.736005    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkgmg\" (UniqueName: \"kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg\") pod \"maas-api-key-cleanup-29894460-7bhb4\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.747867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.736965    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.824866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.824769    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"]
Nov 03 01:00:00.855521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.855416    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"]
Nov 03 01:00:00.858275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858181    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qbvp\" (UniqueName: \"kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.858275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858241    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz6vw\" (UniqueName: \"kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.858498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858292    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9h6\" (UniqueName: \"kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6\") pod \"maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.858498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858378    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.858498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858408    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.858498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858447    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.858498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858492    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.858773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858516    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-7bhb4\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.858773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858552    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.858773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858625    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn4bn\" (UniqueName: \"kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn\") pod \"maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.858773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858682    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxl4x\" (UniqueName: \"kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x\") pod \"maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.858773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858717    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.858773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858773    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xp5z\" (UniqueName: \"kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.859106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858816    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.859106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858846    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkgmg\" (UniqueName: \"kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg\") pod \"maas-api-key-cleanup-29894460-7bhb4\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.859106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858877    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxjzx\" (UniqueName: \"kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx\") pod \"maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.859106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.858954    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.859106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.859039    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.859106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.859078    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw78s\" (UniqueName: \"kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.859419 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.859137    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.862170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.860236    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.862170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.861050    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.862170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.861399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.862170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.861749    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.899388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.899311    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"]
Nov 03 01:00:00.902170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.902134    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.904254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.902786    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.904254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.903387    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-7bhb4\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.904964 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.904644    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.910315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.908888    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qbvp\" (UniqueName: \"kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.910315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.909370    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz6vw\" (UniqueName: \"kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:00.910634 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.910533    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:00.910978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.910803    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:00.912794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.912767    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.913333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.913292    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scppx\" (UniqueName: \"kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:00.918998 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.918940    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9h6\" (UniqueName: \"kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6\") pod \"maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:00.919640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.919620    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"]
Nov 03 01:00:00.919697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.919644    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn4bn\" (UniqueName: \"kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn\") pod \"maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:00.925685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.924353    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxl4x\" (UniqueName: \"kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x\") pod \"maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:00.928788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.928766    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"]
Nov 03 01:00:00.937098 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.936536    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:00.937098 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.937013    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxjzx\" (UniqueName: \"kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx\") pod \"maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:00.938561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.938518    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkgmg\" (UniqueName: \"kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg\") pod \"maas-api-key-cleanup-29894460-7bhb4\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:00.941034 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.940018    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:00.943749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.943727    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xp5z\" (UniqueName: \"kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:00.947848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.946800    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw78s\" (UniqueName: \"kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:00.950084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:00.949996    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"]
Nov 03 01:00:01.022732 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.022703    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:01.032306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.027484    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"]
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.056712    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"]
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.056851    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.070833    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v528v\" (UniqueName: \"kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.070977    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.078725    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.087192    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:01.098638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.096250    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"]
Nov 03 01:00:01.108502 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.102761    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"]
Nov 03 01:00:01.108502 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.104799    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:01.126913 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.114103    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:01.141301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.139000    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:01.155639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.146272    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:01.157551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.156363    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"]
Nov 03 01:00:01.158533 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.157794    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.162001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.161932    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:01.170476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.170403    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"]
Nov 03 01:00:01.170679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.170564    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.171043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.171025    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"]
Nov 03 01:00:01.203993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.202569    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.226514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.204342    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.227071 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.227022    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.227204 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.227134    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrqw\" (UniqueName: \"kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.227292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.227274    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxxg\" (UniqueName: \"kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.227445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.227412    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v528v\" (UniqueName: \"kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.228014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.205188    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.228107 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.207789    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:01.228988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.218219    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:01.229629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.229611    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:01.237529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.237508    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"]
Nov 03 01:00:01.249829 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.249805    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v528v\" (UniqueName: \"kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.274629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.273910    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:01.303315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.303264    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"]
Nov 03 01:00:01.328331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.327586    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"]
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.328831    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.328886    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.328930    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrqw\" (UniqueName: \"kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.329015    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxxg\" (UniqueName: \"kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.329156    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.329238    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrz5n\" (UniqueName: \"kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.331392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.330191    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.332018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.331978    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.344282 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.337133    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"]
Nov 03 01:00:01.344282 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.343020    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.345593 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.345459    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.351542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.351403    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"]
Nov 03 01:00:01.361556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.360104    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"]
Nov 03 01:00:01.380717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.380674    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.388977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.388387    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxxg\" (UniqueName: \"kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.400208 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.400152    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrqw\" (UniqueName: \"kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.410869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.410826    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"]
Nov 03 01:00:01.430562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.430502    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqdxq\" (UniqueName: \"kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq\") pod \"maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.430988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.430965    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6vz\" (UniqueName: \"kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.431224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.431204    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.431417 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.431385    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrz5n\" (UniqueName: \"kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.431597 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.431567    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2wzl\" (UniqueName: \"kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.441603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.441571    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.441870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.440423    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"]
Nov 03 01:00:01.442072 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.442051    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.442244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.442225    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.443001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.442977    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.462736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.457062    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"]
Nov 03 01:00:01.462736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.457092    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrz5n\" (UniqueName: \"kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.472510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.470758    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:01.505477 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.501769    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:01.557479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.544831    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2wzl\" (UniqueName: \"kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.557479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.544884    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.557479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.545056    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.557479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.549338    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.557479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.550645    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.561065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.558156    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:01.561065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.558499    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqdxq\" (UniqueName: \"kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq\") pod \"maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.561065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.558701    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6vz\" (UniqueName: \"kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.561065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.558792    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.561065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.560747    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.580083 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.580042    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2wzl\" (UniqueName: \"kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.582235 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.582205    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqdxq\" (UniqueName: \"kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq\") pod \"maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.585082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.584867    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6vz\" (UniqueName: \"kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.725932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.709577    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:01.725932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.715739    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:01.746764 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.746688    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"]
Nov 03 01:00:01.765670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.765501    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:01.783800 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.783717    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"]
Nov 03 01:00:01.839260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.839185    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"]
Nov 03 01:00:01.845456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.843740    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"]
Nov 03 01:00:01.859182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.854939    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"]
Nov 03 01:00:01.998114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:01.997803    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"]
Nov 03 01:00:02.063807 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.061332    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"]
Nov 03 01:00:02.063807 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.063073    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:02.066505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.065521    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"]
Nov 03 01:00:02.219244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.219164    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerStarted","Data":"ceaf02e82ea903f0e7d30e6df562ea9e69eca4dd3d0a2c912e37de78a5f71559"}
Nov 03 01:00:02.221519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.221481    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerStarted","Data":"1e6c55ec5c3a36d3706b7b7b0f2abe2dedd85011c6e54ad8f53829f5bf39c72e"}
Nov 03 01:00:02.224231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.224210    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerStarted","Data":"b8a0a9f1b5cac9ab171d7ba6199f59d3fc4f0b85f74a21909e88acf6f0deecf8"}
Nov 03 01:00:02.226140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.226066    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" event={"ID":"9e0a52de-9acc-4c3e-bb27-fa6137fee5df","Type":"ContainerStarted","Data":"d84d34e6ca6df5223ae2afaaf90e5af02f292fe5a412ac0e756eb8dc262b7a70"}
Nov 03 01:00:02.231161 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.231101    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerStarted","Data":"b8643cd16e18192114c158f10356387364f3368250a884a063b2b92e03bfd586"}
Nov 03 01:00:02.233250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.233197    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerStarted","Data":"16ded95182c7b679e92a641c1752d51ba77e87895565615891351dd67a18d65c"}
Nov 03 01:00:02.235457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.235293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerStarted","Data":"f88b1f6e1b624f5c492625a1a8a564103aed8f5a800ff7507e1810cb4ad5c2a0"}
Nov 03 01:00:02.283628 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.283595    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"]
Nov 03 01:00:02.333186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.333159    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"]
Nov 03 01:00:02.333471 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:00:02.333407    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a4e5e2_d23b_4006_a3fa_448ded9b054f.slice/crio-87ccddcb4e3a240016db87b9dc5796bb1b23ab03ae38312e4d0234be7cef87bf WatchSource:0}: Error finding container 87ccddcb4e3a240016db87b9dc5796bb1b23ab03ae38312e4d0234be7cef87bf: Status 404 returned error can't find the container with id 87ccddcb4e3a240016db87b9dc5796bb1b23ab03ae38312e4d0234be7cef87bf
Nov 03 01:00:02.396905 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:00:02.396860    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f332463_51f7_4d57_ae51_99f46e0074b6.slice/crio-fe9cb76c787844a3e915b66c23f17763369172bb1ec74c37fd8f13a172c41d02 WatchSource:0}: Error finding container fe9cb76c787844a3e915b66c23f17763369172bb1ec74c37fd8f13a172c41d02: Status 404 returned error can't find the container with id fe9cb76c787844a3e915b66c23f17763369172bb1ec74c37fd8f13a172c41d02
Nov 03 01:00:02.400816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.400781    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"]
Nov 03 01:00:02.467030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.466999    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"]
Nov 03 01:00:02.511218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.511180    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"]
Nov 03 01:00:02.569235 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:00:02.566720    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4171a7bd_72e5_4863_9cdb_5637237dcddf.slice/crio-7c1bc63f8380e129b24700593d87a2c926bb8c88274649835c0d697566d9cba6 WatchSource:0}: Error finding container 7c1bc63f8380e129b24700593d87a2c926bb8c88274649835c0d697566d9cba6: Status 404 returned error can't find the container with id 7c1bc63f8380e129b24700593d87a2c926bb8c88274649835c0d697566d9cba6
Nov 03 01:00:02.583494 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.583452    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"]
Nov 03 01:00:02.600001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.599977    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"]
Nov 03 01:00:02.632090 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.632063    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"]
Nov 03 01:00:02.636791 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.636765    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"]
Nov 03 01:00:02.641553 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.641529    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"]
Nov 03 01:00:02.672190 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.670904    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"]
Nov 03 01:00:02.684625 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.684536    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"]
Nov 03 01:00:02.690090 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:00:02.689636    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a5ddfa_213d_472a_9460_3e770c9cb1a8.slice/crio-fc7d5d91f83bde547209ea8cbca9969300f77d4c4ba705be201aa8dcf968801b WatchSource:0}: Error finding container fc7d5d91f83bde547209ea8cbca9969300f77d4c4ba705be201aa8dcf968801b: Status 404 returned error can't find the container with id fc7d5d91f83bde547209ea8cbca9969300f77d4c4ba705be201aa8dcf968801b
Nov 03 01:00:02.691048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.691014    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"]
Nov 03 01:00:02.712757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.712725    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"]
Nov 03 01:00:02.742621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.742591    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"]
Nov 03 01:00:02.942694 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.942672    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 01:00:02.943108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:02.942757    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 01:00:03.246575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.246541    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerStarted","Data":"fc7d5d91f83bde547209ea8cbca9969300f77d4c4ba705be201aa8dcf968801b"}
Nov 03 01:00:03.248013 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.247991    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" event={"ID":"fd75e99a-1821-4bb0-8f2e-16ce5c44db28","Type":"ContainerStarted","Data":"aa77f74058e569f7505f0a4d30a314504fec3d09349315ffa31bbe842b46dfe5"}
Nov 03 01:00:03.249285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.249268    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerStarted","Data":"e1b75de87a3241f525c00309c9bcd113571346950036eeadc892bac692e6f6fb"}
Nov 03 01:00:03.250683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.250653    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerStarted","Data":"30354669139d39011abda1857909ae48e1b215c7b48c38637a02862735b13abe"}
Nov 03 01:00:03.251994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.251968    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" event={"ID":"aa394189-738e-4dc7-8cef-c0576e53a98d","Type":"ContainerStarted","Data":"d4a94d8999cceeecee1298504ac66004e614ec57c6baaa14aa677419f2a9a6a9"}
Nov 03 01:00:03.253537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.253516    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerStarted","Data":"c526881265ac7a6896e923588feca44dc4ecbf7f1f03930f596fcad26753fbf4"}
Nov 03 01:00:03.255102 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.255080    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerStarted","Data":"17ba1e28fb49bcbff490a5c5d6d2894c691b0b2457f76b0e74dee3c2247ac1ed"}
Nov 03 01:00:03.261919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.261893    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" event={"ID":"90507494-4818-4eb4-a1cb-a3e682b3bdd8","Type":"ContainerStarted","Data":"7808f577dd239e100183785b9b37a9e5feb60725378c06547758a5e1be2e2bfe"}
Nov 03 01:00:03.264792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.264768    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" event={"ID":"36a4e5e2-d23b-4006-a3fa-448ded9b054f","Type":"ContainerStarted","Data":"87ccddcb4e3a240016db87b9dc5796bb1b23ab03ae38312e4d0234be7cef87bf"}
Nov 03 01:00:03.266293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.266274    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerStarted","Data":"0745e07993000737168c5bf6637945b015451dc7994d6b0e18429b13492f20fa"}
Nov 03 01:00:03.268405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.268377    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" event={"ID":"03449032-8806-4415-a9ff-7ac5f37df4ab","Type":"ContainerStarted","Data":"6dac84ea238a9703105ef13f6a5b33b7b48b371ee5b33192cf4e6fe151325e26"}
Nov 03 01:00:03.270483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.270459    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerStarted","Data":"3eed94e313393a7ace836ba4554c1965536f32bf643d60eb339d9f3a7ffd67b1"}
Nov 03 01:00:03.274630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.272014    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerStarted","Data":"8cce9b93a68bdb1b70d77f2fc5b11f46d70bd3c47e327d68a7675432f6a8cfd8"}
Nov 03 01:00:03.274630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.274073    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" event={"ID":"4171a7bd-72e5-4863-9cdb-5637237dcddf","Type":"ContainerStarted","Data":"7c1bc63f8380e129b24700593d87a2c926bb8c88274649835c0d697566d9cba6"}
Nov 03 01:00:03.275500 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.275481    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerStarted","Data":"dba986e1ee6092e2c7130e250ebe8d372bd3b088351a7d1c76728f8169a44094"}
Nov 03 01:00:03.275571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.275514    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerStarted","Data":"7c4f82b713810fea7cc051aa9539734d2d1828d38bb06b81a3c0309221731758"}
Nov 03 01:00:03.277326 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.277304    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" event={"ID":"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab","Type":"ContainerStarted","Data":"fb81cc62f409d9cda6643ef403d5b2638c3f82e28f84e904e56dd6438ddae22f"}
Nov 03 01:00:03.278861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.278839    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerStarted","Data":"ff646c25fd724d3d8bf6425f308682fb0b0ee9a8f777ee7e91d44ab320753b6c"}
Nov 03 01:00:03.280319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.280301    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" event={"ID":"8f332463-51f7-4d57-ae51-99f46e0074b6","Type":"ContainerStarted","Data":"fe9cb76c787844a3e915b66c23f17763369172bb1ec74c37fd8f13a172c41d02"}
Nov 03 01:00:03.281931 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.281913    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerStarted","Data":"756723d3c96ec6a54b608c940d94e64a5c072cb5bb1b48bd47a91af7b23b31dd"}
Nov 03 01:00:03.284390 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.284368    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerStarted","Data":"0d8ce831b3612c0194e82e403d63f2483eb4e9c7853ebe26de156bb9c723f020"}
Nov 03 01:00:03.287344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.287321    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" event={"ID":"9e0a52de-9acc-4c3e-bb27-fa6137fee5df","Type":"ContainerStarted","Data":"6f0666d1f08805e04702ba8105445252cb5aca91bb703bafa88993a992c62923"}
Nov 03 01:00:03.288784 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.288764    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" event={"ID":"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c","Type":"ContainerStarted","Data":"21878433db2f18a300427880122ff172f692123632674ced8bfd25c492829cf1"}
Nov 03 01:00:03.290405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.290383    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerStarted","Data":"a78e048631dc97261cac0dcd61296d6f3a758ecb96b26ce95b97f55d51910cc9"}
Nov 03 01:00:03.296117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.296073    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" podStartSLOduration=3.296060406 podStartE2EDuration="3.296060406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.294780651 +0000 UTC m=+3285.616317481" watchObservedRunningTime="2026-11-03 01:00:03.296060406 +0000 UTC m=+3285.617597206"
Nov 03 01:00:03.340259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.340145    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" podStartSLOduration=3.340125048 podStartE2EDuration="3.340125048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.333861296 +0000 UTC m=+3285.655398120" watchObservedRunningTime="2026-11-03 01:00:03.340125048 +0000 UTC m=+3285.661661853"
Nov 03 01:00:03.378162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.375696    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" podStartSLOduration=3.375676609 podStartE2EDuration="3.375676609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.372732005 +0000 UTC m=+3285.694268832" watchObservedRunningTime="2026-11-03 01:00:03.375676609 +0000 UTC m=+3285.697213410"
Nov 03 01:00:03.457535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.420413    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" podStartSLOduration=3.420399528 podStartE2EDuration="3.420399528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.417239885 +0000 UTC m=+3285.738776686" watchObservedRunningTime="2026-11-03 01:00:03.420399528 +0000 UTC m=+3285.741936329"
Nov 03 01:00:03.463143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.463080    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" podStartSLOduration=3.4630598 podStartE2EDuration="3.4630598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.460379555 +0000 UTC m=+3285.781916379" watchObservedRunningTime="2026-11-03 01:00:03.4630598 +0000 UTC m=+3285.784596601"
Nov 03 01:00:03.504961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.504912    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" podStartSLOduration=3.504898779 podStartE2EDuration="3.504898779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.500906218 +0000 UTC m=+3285.822443021" watchObservedRunningTime="2026-11-03 01:00:03.504898779 +0000 UTC m=+3285.826435578"
Nov 03 01:00:03.548046 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.547995    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" podStartSLOduration=3.547979948 podStartE2EDuration="3.547979948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.543707388 +0000 UTC m=+3285.865244207" watchObservedRunningTime="2026-11-03 01:00:03.547979948 +0000 UTC m=+3285.869516747"
Nov 03 01:00:03.586292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.586231    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" podStartSLOduration=3.586211842 podStartE2EDuration="3.586211842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.582218794 +0000 UTC m=+3285.903755595" watchObservedRunningTime="2026-11-03 01:00:03.586211842 +0000 UTC m=+3285.907748644"
Nov 03 01:00:03.914769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:03.914739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:04.041179 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.041108    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"]
Nov 03 01:00:04.046325 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.046298    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d748c2-zwhxd"]
Nov 03 01:00:04.303698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.303655    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerStarted","Data":"59ef4517d9bdefedf8c6f1961767912d79346f8a1bdc83954f1d1de1b3263919"}
Nov 03 01:00:04.305849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.305821    4550 generic.go:356] "Generic (PLEG): container finished" podID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerID="f5e9f03b2578288bfce9bda4bd2e1e16c277b7eb7775015b3a8c422a350e221c" exitCode=60
Nov 03 01:00:04.305956 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.305909    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" event={"ID":"03449032-8806-4415-a9ff-7ac5f37df4ab","Type":"ContainerDied","Data":"f5e9f03b2578288bfce9bda4bd2e1e16c277b7eb7775015b3a8c422a350e221c"}
Nov 03 01:00:04.306267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.306247    4550 scope.go:122] "RemoveContainer" containerID="f5e9f03b2578288bfce9bda4bd2e1e16c277b7eb7775015b3a8c422a350e221c"
Nov 03 01:00:04.309237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.309215    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerStarted","Data":"1db6ad126af8d25bf3776c215b38446b8f4d2db3453c42e6e89baf1beb8dcc79"}
Nov 03 01:00:04.311328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.311130    4550 generic.go:356] "Generic (PLEG): container finished" podID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerID="8cce9b93a68bdb1b70d77f2fc5b11f46d70bd3c47e327d68a7675432f6a8cfd8" exitCode=60
Nov 03 01:00:04.311328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.311201    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerDied","Data":"8cce9b93a68bdb1b70d77f2fc5b11f46d70bd3c47e327d68a7675432f6a8cfd8"}
Nov 03 01:00:04.311568 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.311518    4550 scope.go:122] "RemoveContainer" containerID="8cce9b93a68bdb1b70d77f2fc5b11f46d70bd3c47e327d68a7675432f6a8cfd8"
Nov 03 01:00:04.312896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.312873    4550 generic.go:356] "Generic (PLEG): container finished" podID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerID="5e549c46aed23e617a1d94f42f9fa77831825871d0c0107ae17910f5c3f71718" exitCode=60
Nov 03 01:00:04.313005 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.312950    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" event={"ID":"4171a7bd-72e5-4863-9cdb-5637237dcddf","Type":"ContainerDied","Data":"5e549c46aed23e617a1d94f42f9fa77831825871d0c0107ae17910f5c3f71718"}
Nov 03 01:00:04.313147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.313129    4550 scope.go:122] "RemoveContainer" containerID="5e549c46aed23e617a1d94f42f9fa77831825871d0c0107ae17910f5c3f71718"
Nov 03 01:00:04.314727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.314707    4550 generic.go:356] "Generic (PLEG): container finished" podID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerID="dba986e1ee6092e2c7130e250ebe8d372bd3b088351a7d1c76728f8169a44094" exitCode=60
Nov 03 01:00:04.314849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.314798    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerDied","Data":"dba986e1ee6092e2c7130e250ebe8d372bd3b088351a7d1c76728f8169a44094"}
Nov 03 01:00:04.315022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.315001    4550 scope.go:122] "RemoveContainer" containerID="dba986e1ee6092e2c7130e250ebe8d372bd3b088351a7d1c76728f8169a44094"
Nov 03 01:00:04.316616 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.316593    4550 generic.go:356] "Generic (PLEG): container finished" podID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerID="7131f0c21a531b8c10d3b7225deff88f05ec543c3f3b22c98478fc37987b9fda" exitCode=60
Nov 03 01:00:04.316720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.316621    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" event={"ID":"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab","Type":"ContainerDied","Data":"7131f0c21a531b8c10d3b7225deff88f05ec543c3f3b22c98478fc37987b9fda"}
Nov 03 01:00:04.316884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.316867    4550 scope.go:122] "RemoveContainer" containerID="7131f0c21a531b8c10d3b7225deff88f05ec543c3f3b22c98478fc37987b9fda"
Nov 03 01:00:04.318519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.318496    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerStarted","Data":"914a7cc3951a9f37fe8c4b4d88381873cf5f1ccfd4599cd9234f49d36410a1ec"}
Nov 03 01:00:04.320551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.320531    4550 generic.go:356] "Generic (PLEG): container finished" podID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerID="8f8aad49b00ea9b437f31650188b7c2096a0547f9682374d82a737fd48d0ccfc" exitCode=60
Nov 03 01:00:04.320653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.320616    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" event={"ID":"8f332463-51f7-4d57-ae51-99f46e0074b6","Type":"ContainerDied","Data":"8f8aad49b00ea9b437f31650188b7c2096a0547f9682374d82a737fd48d0ccfc"}
Nov 03 01:00:04.320859 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.320840    4550 scope.go:122] "RemoveContainer" containerID="8f8aad49b00ea9b437f31650188b7c2096a0547f9682374d82a737fd48d0ccfc"
Nov 03 01:00:04.327054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.327030    4550 generic.go:356] "Generic (PLEG): container finished" podID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerID="756723d3c96ec6a54b608c940d94e64a5c072cb5bb1b48bd47a91af7b23b31dd" exitCode=60
Nov 03 01:00:04.327177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.327143    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerDied","Data":"756723d3c96ec6a54b608c940d94e64a5c072cb5bb1b48bd47a91af7b23b31dd"}
Nov 03 01:00:04.327374 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.327335    4550 scope.go:122] "RemoveContainer" containerID="756723d3c96ec6a54b608c940d94e64a5c072cb5bb1b48bd47a91af7b23b31dd"
Nov 03 01:00:04.329541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.329291    4550 generic.go:356] "Generic (PLEG): container finished" podID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerID="0d8ce831b3612c0194e82e403d63f2483eb4e9c7853ebe26de156bb9c723f020" exitCode=60
Nov 03 01:00:04.329541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.329353    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerDied","Data":"0d8ce831b3612c0194e82e403d63f2483eb4e9c7853ebe26de156bb9c723f020"}
Nov 03 01:00:04.329729 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.329712    4550 scope.go:122] "RemoveContainer" containerID="0d8ce831b3612c0194e82e403d63f2483eb4e9c7853ebe26de156bb9c723f020"
Nov 03 01:00:04.330949 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.330931    4550 generic.go:356] "Generic (PLEG): container finished" podID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerID="6f0666d1f08805e04702ba8105445252cb5aca91bb703bafa88993a992c62923" exitCode=60
Nov 03 01:00:04.331038 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.331008    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" event={"ID":"9e0a52de-9acc-4c3e-bb27-fa6137fee5df","Type":"ContainerDied","Data":"6f0666d1f08805e04702ba8105445252cb5aca91bb703bafa88993a992c62923"}
Nov 03 01:00:04.331227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.331209    4550 scope.go:122] "RemoveContainer" containerID="6f0666d1f08805e04702ba8105445252cb5aca91bb703bafa88993a992c62923"
Nov 03 01:00:04.332819 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.332798    4550 generic.go:356] "Generic (PLEG): container finished" podID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerID="22e07ada143c0524037d4269fa5f248f097c96281203677a73d0fcc71ff58b3d" exitCode=60
Nov 03 01:00:04.332934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.332882    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" event={"ID":"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c","Type":"ContainerDied","Data":"22e07ada143c0524037d4269fa5f248f097c96281203677a73d0fcc71ff58b3d"}
Nov 03 01:00:04.333053 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.333041    4550 scope.go:122] "RemoveContainer" containerID="22e07ada143c0524037d4269fa5f248f097c96281203677a73d0fcc71ff58b3d"
Nov 03 01:00:04.334576 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.334557    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9598928-b49d-40ab-9e70-e127b122c329" containerID="a78e048631dc97261cac0dcd61296d6f3a758ecb96b26ce95b97f55d51910cc9" exitCode=60
Nov 03 01:00:04.334763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.334599    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerDied","Data":"a78e048631dc97261cac0dcd61296d6f3a758ecb96b26ce95b97f55d51910cc9"}
Nov 03 01:00:04.335292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.335267    4550 scope.go:122] "RemoveContainer" containerID="a78e048631dc97261cac0dcd61296d6f3a758ecb96b26ce95b97f55d51910cc9"
Nov 03 01:00:04.339237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.339215    4550 generic.go:356] "Generic (PLEG): container finished" podID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerID="344452d376c55529a85bdcdef0277f26b80194f4bf720c8d8e7b5eed83591ee4" exitCode=60
Nov 03 01:00:04.339365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.339291    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerDied","Data":"344452d376c55529a85bdcdef0277f26b80194f4bf720c8d8e7b5eed83591ee4"}
Nov 03 01:00:04.339686 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.339667    4550 scope.go:122] "RemoveContainer" containerID="344452d376c55529a85bdcdef0277f26b80194f4bf720c8d8e7b5eed83591ee4"
Nov 03 01:00:04.352801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.352756    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" podStartSLOduration=4.352738996 podStartE2EDuration="4.352738996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.348981299 +0000 UTC m=+3286.670518098" watchObservedRunningTime="2026-11-03 01:00:04.352738996 +0000 UTC m=+3286.674275799"
Nov 03 01:00:04.353139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.353120    4550 generic.go:356] "Generic (PLEG): container finished" podID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerID="8b668b0f94042058e603bbd410ae6b2800b2b7721fbe411d87f7c57957151867" exitCode=60
Nov 03 01:00:04.353665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.353260    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" event={"ID":"fd75e99a-1821-4bb0-8f2e-16ce5c44db28","Type":"ContainerDied","Data":"8b668b0f94042058e603bbd410ae6b2800b2b7721fbe411d87f7c57957151867"}
Nov 03 01:00:04.353821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.353549    4550 scope.go:122] "RemoveContainer" containerID="8b668b0f94042058e603bbd410ae6b2800b2b7721fbe411d87f7c57957151867"
Nov 03 01:00:04.362696 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.362668    4550 generic.go:356] "Generic (PLEG): container finished" podID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerID="92f2779cf857a072b50207dbc5768c2f9df59855b57fb965ce99568362394f15" exitCode=60
Nov 03 01:00:04.362898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.362739    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerDied","Data":"92f2779cf857a072b50207dbc5768c2f9df59855b57fb965ce99568362394f15"}
Nov 03 01:00:04.363186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.363163    4550 scope.go:122] "RemoveContainer" containerID="92f2779cf857a072b50207dbc5768c2f9df59855b57fb965ce99568362394f15"
Nov 03 01:00:04.366168 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.366148    4550 generic.go:356] "Generic (PLEG): container finished" podID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerID="9666f4f05e52f876ca137856d34838d5a7eac35ef9077deb2b4de88aeda87f79" exitCode=60
Nov 03 01:00:04.366251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.366224    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerDied","Data":"9666f4f05e52f876ca137856d34838d5a7eac35ef9077deb2b4de88aeda87f79"}
Nov 03 01:00:04.366650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.366621    4550 scope.go:122] "RemoveContainer" containerID="9666f4f05e52f876ca137856d34838d5a7eac35ef9077deb2b4de88aeda87f79"
Nov 03 01:00:04.368535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.368513    4550 generic.go:356] "Generic (PLEG): container finished" podID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerID="b71e120738262766e780e28980d7260af2357bbf65155f207fd99b47382bc9e5" exitCode=60
Nov 03 01:00:04.368638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.368593    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" event={"ID":"aa394189-738e-4dc7-8cef-c0576e53a98d","Type":"ContainerDied","Data":"b71e120738262766e780e28980d7260af2357bbf65155f207fd99b47382bc9e5"}
Nov 03 01:00:04.368908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.368891    4550 scope.go:122] "RemoveContainer" containerID="b71e120738262766e780e28980d7260af2357bbf65155f207fd99b47382bc9e5"
Nov 03 01:00:04.372230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.372038    4550 generic.go:356] "Generic (PLEG): container finished" podID="4ca04320-6331-4640-b213-c761707a2024" containerID="c526881265ac7a6896e923588feca44dc4ecbf7f1f03930f596fcad26753fbf4" exitCode=60
Nov 03 01:00:04.372230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.372189    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerDied","Data":"c526881265ac7a6896e923588feca44dc4ecbf7f1f03930f596fcad26753fbf4"}
Nov 03 01:00:04.372468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.372416    4550 scope.go:122] "RemoveContainer" containerID="c526881265ac7a6896e923588feca44dc4ecbf7f1f03930f596fcad26753fbf4"
Nov 03 01:00:04.390797 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.390679    4550 generic.go:356] "Generic (PLEG): container finished" podID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerID="17ba1e28fb49bcbff490a5c5d6d2894c691b0b2457f76b0e74dee3c2247ac1ed" exitCode=60
Nov 03 01:00:04.390928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.390863    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerDied","Data":"17ba1e28fb49bcbff490a5c5d6d2894c691b0b2457f76b0e74dee3c2247ac1ed"}
Nov 03 01:00:04.391423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.391332    4550 scope.go:122] "RemoveContainer" containerID="17ba1e28fb49bcbff490a5c5d6d2894c691b0b2457f76b0e74dee3c2247ac1ed"
Nov 03 01:00:04.408930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.408897    4550 generic.go:356] "Generic (PLEG): container finished" podID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerID="4b9a1ca8a34dc9a9c8c3e57a88703850ff0986992b8ba756f1436dba829ca303" exitCode=60
Nov 03 01:00:04.409042 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.409022    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" event={"ID":"90507494-4818-4eb4-a1cb-a3e682b3bdd8","Type":"ContainerDied","Data":"4b9a1ca8a34dc9a9c8c3e57a88703850ff0986992b8ba756f1436dba829ca303"}
Nov 03 01:00:04.435801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.435662    4550 generic.go:356] "Generic (PLEG): container finished" podID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerID="2cab6e0ecfb597a91f61c173f95728a1779dff1ee4ca5602c0d7e60a2b9b8cf4" exitCode=60
Nov 03 01:00:04.435801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.435749    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" event={"ID":"36a4e5e2-d23b-4006-a3fa-448ded9b054f","Type":"ContainerDied","Data":"2cab6e0ecfb597a91f61c173f95728a1779dff1ee4ca5602c0d7e60a2b9b8cf4"}
Nov 03 01:00:04.436728 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.436703    4550 scope.go:122] "RemoveContainer" containerID="4b9a1ca8a34dc9a9c8c3e57a88703850ff0986992b8ba756f1436dba829ca303"
Nov 03 01:00:04.438107 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.438083    4550 scope.go:122] "RemoveContainer" containerID="2cab6e0ecfb597a91f61c173f95728a1779dff1ee4ca5602c0d7e60a2b9b8cf4"
Nov 03 01:00:04.633147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.633087    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" podStartSLOduration=4.633072262 podStartE2EDuration="4.633072262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.630564305 +0000 UTC m=+3286.952101124" watchObservedRunningTime="2026-11-03 01:00:04.633072262 +0000 UTC m=+3286.954609090"
Nov 03 01:00:04.800416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:04.800344    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" podStartSLOduration=4.80032359 podStartE2EDuration="4.80032359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.792423442 +0000 UTC m=+3287.113960267" watchObservedRunningTime="2026-11-03 01:00:04.80032359 +0000 UTC m=+3287.121860391"
Nov 03 01:00:05.453996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.453966    4550 generic.go:356] "Generic (PLEG): container finished" podID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerID="59ef4517d9bdefedf8c6f1961767912d79346f8a1bdc83954f1d1de1b3263919" exitCode=60
Nov 03 01:00:05.454528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.454051    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerDied","Data":"59ef4517d9bdefedf8c6f1961767912d79346f8a1bdc83954f1d1de1b3263919"}
Nov 03 01:00:05.454528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.454503    4550 scope.go:122] "RemoveContainer" containerID="59ef4517d9bdefedf8c6f1961767912d79346f8a1bdc83954f1d1de1b3263919"
Nov 03 01:00:05.456773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.456751    4550 generic.go:356] "Generic (PLEG): container finished" podID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerID="1db6ad126af8d25bf3776c215b38446b8f4d2db3453c42e6e89baf1beb8dcc79" exitCode=60
Nov 03 01:00:05.456908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.456866    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerDied","Data":"1db6ad126af8d25bf3776c215b38446b8f4d2db3453c42e6e89baf1beb8dcc79"}
Nov 03 01:00:05.457226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.457198    4550 scope.go:122] "RemoveContainer" containerID="1db6ad126af8d25bf3776c215b38446b8f4d2db3453c42e6e89baf1beb8dcc79"
Nov 03 01:00:05.461992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.461803    4550 generic.go:356] "Generic (PLEG): container finished" podID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerID="914a7cc3951a9f37fe8c4b4d88381873cf5f1ccfd4599cd9234f49d36410a1ec" exitCode=60
Nov 03 01:00:05.461992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.461928    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerDied","Data":"914a7cc3951a9f37fe8c4b4d88381873cf5f1ccfd4599cd9234f49d36410a1ec"}
Nov 03 01:00:05.462158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.462123    4550 scope.go:122] "RemoveContainer" containerID="914a7cc3951a9f37fe8c4b4d88381873cf5f1ccfd4599cd9234f49d36410a1ec"
Nov 03 01:00:05.465756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.464333    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" event={"ID":"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c","Type":"ContainerStarted","Data":"70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e"}
Nov 03 01:00:05.469780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.468182    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerStarted","Data":"d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130"}
Nov 03 01:00:05.472111 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.472077    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerStarted","Data":"9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f"}
Nov 03 01:00:05.474892 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.474835    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerStarted","Data":"acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306"}
Nov 03 01:00:05.477839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.477419    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerStarted","Data":"a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4"}
Nov 03 01:00:05.490755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.490081    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerStarted","Data":"2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5"}
Nov 03 01:00:05.541927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.541869    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" podStartSLOduration=5.541853256 podStartE2EDuration="5.541853256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.540134925 +0000 UTC m=+3287.861671725" watchObservedRunningTime="2026-11-03 01:00:05.541853256 +0000 UTC m=+3287.863390056"
Nov 03 01:00:05.595852 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.595790    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" podStartSLOduration=5.59577416 podStartE2EDuration="5.59577416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.593747209 +0000 UTC m=+3287.915304124" watchObservedRunningTime="2026-11-03 01:00:05.59577416 +0000 UTC m=+3287.917310957"
Nov 03 01:00:05.689328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.689282    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" podStartSLOduration=5.689264771 podStartE2EDuration="5.689264771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.688425358 +0000 UTC m=+3288.009962169" watchObservedRunningTime="2026-11-03 01:00:05.689264771 +0000 UTC m=+3288.010801571"
Nov 03 01:00:05.768676 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.768327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:05.851105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.851062    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8411618-78bf-4398-b7f4-3326495a8691" path="/var/lib/kubelet/pods/d8411618-78bf-4398-b7f4-3326495a8691/volumes"
Nov 03 01:00:05.858008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:05.853658    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" podStartSLOduration=5.853638649 podStartE2EDuration="5.853638649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.852549576 +0000 UTC m=+3288.174086405" watchObservedRunningTime="2026-11-03 01:00:05.853638649 +0000 UTC m=+3288.175175454"
Nov 03 01:00:06.502894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.502851    4550 generic.go:356] "Generic (PLEG): container finished" podID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerID="edabe2f374efe1b1762378d4b6a71c70bcdcd3885707be5449b230222c0d0edd" exitCode=60
Nov 03 01:00:06.503388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.503181    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" event={"ID":"aa394189-738e-4dc7-8cef-c0576e53a98d","Type":"ContainerDied","Data":"edabe2f374efe1b1762378d4b6a71c70bcdcd3885707be5449b230222c0d0edd"}
Nov 03 01:00:06.503388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.503249    4550 scope.go:122] "RemoveContainer" containerID="b71e120738262766e780e28980d7260af2357bbf65155f207fd99b47382bc9e5"
Nov 03 01:00:06.503531 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.503511    4550 scope.go:122] "RemoveContainer" containerID="edabe2f374efe1b1762378d4b6a71c70bcdcd3885707be5449b230222c0d0edd"
Nov 03 01:00:06.504769 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.504485    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj_odh-ai-gateway-infra(aa394189-738e-4dc7-8cef-c0576e53a98d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d"
Nov 03 01:00:06.507290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.506369    4550 generic.go:356] "Generic (PLEG): container finished" podID="4ca04320-6331-4640-b213-c761707a2024" containerID="d69e30e9ca87ea4940e5e150fc9f080a2f57564bc0de49df7122b778f65d686e" exitCode=60
Nov 03 01:00:06.507290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.506397    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerDied","Data":"d69e30e9ca87ea4940e5e150fc9f080a2f57564bc0de49df7122b778f65d686e"}
Nov 03 01:00:06.507290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.506822    4550 scope.go:122] "RemoveContainer" containerID="d69e30e9ca87ea4940e5e150fc9f080a2f57564bc0de49df7122b778f65d686e"
Nov 03 01:00:06.507290 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.507244    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d_odh-ai-gateway-infra(4ca04320-6331-4640-b213-c761707a2024)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" podUID="4ca04320-6331-4640-b213-c761707a2024"
Nov 03 01:00:06.509018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.508998    4550 generic.go:356] "Generic (PLEG): container finished" podID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerID="2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5" exitCode=60
Nov 03 01:00:06.509226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.509030    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerDied","Data":"2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5"}
Nov 03 01:00:06.509483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.509422    4550 scope.go:122] "RemoveContainer" containerID="2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5"
Nov 03 01:00:06.509853 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.509827    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-7bhb4_odh-ai-gateway-infra(c3442fb4-b070-4d02-b0a7-0f78e83f5c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61"
Nov 03 01:00:06.511906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.511878    4550 generic.go:356] "Generic (PLEG): container finished" podID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerID="6846393c7746d6d8c8e0ffba0e22d3b695b912812a2aa9f2c60b181746b0b436" exitCode=60
Nov 03 01:00:06.512012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.511969    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" event={"ID":"90507494-4818-4eb4-a1cb-a3e682b3bdd8","Type":"ContainerDied","Data":"6846393c7746d6d8c8e0ffba0e22d3b695b912812a2aa9f2c60b181746b0b436"}
Nov 03 01:00:06.512470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.512409    4550 scope.go:122] "RemoveContainer" containerID="6846393c7746d6d8c8e0ffba0e22d3b695b912812a2aa9f2c60b181746b0b436"
Nov 03 01:00:06.512809 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.512784    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx_odh-ai-gateway-infra(90507494-4818-4eb4-a1cb-a3e682b3bdd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8"
Nov 03 01:00:06.515272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.515250    4550 generic.go:356] "Generic (PLEG): container finished" podID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerID="6d51ee38a9656a9b5dd7623b5f150a0537bb091a81be377d0aa99e949daf5b23" exitCode=60
Nov 03 01:00:06.515388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.515325    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" event={"ID":"36a4e5e2-d23b-4006-a3fa-448ded9b054f","Type":"ContainerDied","Data":"6d51ee38a9656a9b5dd7623b5f150a0537bb091a81be377d0aa99e949daf5b23"}
Nov 03 01:00:06.515940 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.515619    4550 scope.go:122] "RemoveContainer" containerID="6d51ee38a9656a9b5dd7623b5f150a0537bb091a81be377d0aa99e949daf5b23"
Nov 03 01:00:06.515940 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.515903    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh_odh-ai-gateway-infra(36a4e5e2-d23b-4006-a3fa-448ded9b054f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f"
Nov 03 01:00:06.517929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.517903    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerStarted","Data":"39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60"}
Nov 03 01:00:06.522128 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.522103    4550 generic.go:356] "Generic (PLEG): container finished" podID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerID="d6b94d1d651340b907b155805006034ebf3bbdf46f166d718203b8f38b416035" exitCode=60
Nov 03 01:00:06.522227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.522177    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" event={"ID":"03449032-8806-4415-a9ff-7ac5f37df4ab","Type":"ContainerDied","Data":"d6b94d1d651340b907b155805006034ebf3bbdf46f166d718203b8f38b416035"}
Nov 03 01:00:06.522441 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.522411    4550 scope.go:122] "RemoveContainer" containerID="d6b94d1d651340b907b155805006034ebf3bbdf46f166d718203b8f38b416035"
Nov 03 01:00:06.522729 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.522705    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc_odh-ai-gateway-infra(03449032-8806-4415-a9ff-7ac5f37df4ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab"
Nov 03 01:00:06.524946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.524618    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerStarted","Data":"1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849"}
Nov 03 01:00:06.526909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.526881    4550 generic.go:356] "Generic (PLEG): container finished" podID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerID="61ccc9a8e8f854a5d31cd113b5f33ec1fa36575a4e2aa13fed94c124221609fb" exitCode=60
Nov 03 01:00:06.526999 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.526944    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerDied","Data":"61ccc9a8e8f854a5d31cd113b5f33ec1fa36575a4e2aa13fed94c124221609fb"}
Nov 03 01:00:06.527303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.527286    4550 scope.go:122] "RemoveContainer" containerID="61ccc9a8e8f854a5d31cd113b5f33ec1fa36575a4e2aa13fed94c124221609fb"
Nov 03 01:00:06.527615 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.527582    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c_odh-ai-gateway-infra(30c9afba-79f5-41d3-ae1e-0cf85863e6d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6"
Nov 03 01:00:06.530105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.529667    4550 generic.go:356] "Generic (PLEG): container finished" podID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerID="790acc2e378b37299c695f0fc3e4bb7974abc5a770fd1def0aca63de923dd655" exitCode=60
Nov 03 01:00:06.530105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.529745    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" event={"ID":"4171a7bd-72e5-4863-9cdb-5637237dcddf","Type":"ContainerDied","Data":"790acc2e378b37299c695f0fc3e4bb7974abc5a770fd1def0aca63de923dd655"}
Nov 03 01:00:06.530105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.529944    4550 scope.go:122] "RemoveContainer" containerID="790acc2e378b37299c695f0fc3e4bb7974abc5a770fd1def0aca63de923dd655"
Nov 03 01:00:06.530307 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.530195    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx_odh-ai-gateway-infra(4171a7bd-72e5-4863-9cdb-5637237dcddf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf"
Nov 03 01:00:06.531829 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.531810    4550 generic.go:356] "Generic (PLEG): container finished" podID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerID="52a2c6db53b3e99542319888946cd4af68b467b949aaad70fb625875c30433fb" exitCode=60
Nov 03 01:00:06.531948 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.531893    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerDied","Data":"52a2c6db53b3e99542319888946cd4af68b467b949aaad70fb625875c30433fb"}
Nov 03 01:00:06.532128 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.532111    4550 scope.go:122] "RemoveContainer" containerID="52a2c6db53b3e99542319888946cd4af68b467b949aaad70fb625875c30433fb"
Nov 03 01:00:06.532375 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.532354    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6_odh-ai-gateway-infra(1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce"
Nov 03 01:00:06.533897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.533875    4550 generic.go:356] "Generic (PLEG): container finished" podID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerID="c19d4a9fee7405c1a5249f75a192ed58598858438402938a87b7f87b8b5ff634" exitCode=60
Nov 03 01:00:06.533997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.533951    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" event={"ID":"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab","Type":"ContainerDied","Data":"c19d4a9fee7405c1a5249f75a192ed58598858438402938a87b7f87b8b5ff634"}
Nov 03 01:00:06.534185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.534168    4550 scope.go:122] "RemoveContainer" containerID="c19d4a9fee7405c1a5249f75a192ed58598858438402938a87b7f87b8b5ff634"
Nov 03 01:00:06.534446 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.534411    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp_odh-ai-gateway-infra(620eee4a-a44a-45ce-8cdc-b9a4368aa6ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab"
Nov 03 01:00:06.535761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.535734    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerStarted","Data":"8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad"}
Nov 03 01:00:06.537926 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.537905    4550 generic.go:356] "Generic (PLEG): container finished" podID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerID="0b3e2f751cf36128879a8a713b459fbe1637a95d7834682268a2347cad8a3c43" exitCode=60
Nov 03 01:00:06.538014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.537986    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" event={"ID":"8f332463-51f7-4d57-ae51-99f46e0074b6","Type":"ContainerDied","Data":"0b3e2f751cf36128879a8a713b459fbe1637a95d7834682268a2347cad8a3c43"}
Nov 03 01:00:06.538185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.538168    4550 scope.go:122] "RemoveContainer" containerID="0b3e2f751cf36128879a8a713b459fbe1637a95d7834682268a2347cad8a3c43"
Nov 03 01:00:06.538466 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.538408    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n_odh-ai-gateway-infra(8f332463-51f7-4d57-ae51-99f46e0074b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6"
Nov 03 01:00:06.539967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.539939    4550 generic.go:356] "Generic (PLEG): container finished" podID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerID="2a057aea2252a251ca4f11b8b3a70f09921304d3e3b47c54d0abd9d19bab6f4a" exitCode=60
Nov 03 01:00:06.540066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.540024    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerDied","Data":"2a057aea2252a251ca4f11b8b3a70f09921304d3e3b47c54d0abd9d19bab6f4a"}
Nov 03 01:00:06.540320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.540243    4550 scope.go:122] "RemoveContainer" containerID="2a057aea2252a251ca4f11b8b3a70f09921304d3e3b47c54d0abd9d19bab6f4a"
Nov 03 01:00:06.540534 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.540509    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b_odh-ai-gateway-infra(3117c03f-2cc4-467a-8716-fb4e1b6c0f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e"
Nov 03 01:00:06.542505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.542485    4550 generic.go:356] "Generic (PLEG): container finished" podID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerID="04fb01e23017d6e60104701f45cdddbbe1af5def81eafd7217dd9c9b471e75e4" exitCode=60
Nov 03 01:00:06.542609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.542510    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerDied","Data":"04fb01e23017d6e60104701f45cdddbbe1af5def81eafd7217dd9c9b471e75e4"}
Nov 03 01:00:06.542799 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.542782    4550 scope.go:122] "RemoveContainer" containerID="04fb01e23017d6e60104701f45cdddbbe1af5def81eafd7217dd9c9b471e75e4"
Nov 03 01:00:06.543002 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.542981    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc_odh-ai-gateway-infra(deec6453-23fd-49d0-aded-4518c0e1b5e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2"
Nov 03 01:00:06.544628 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.544607    4550 generic.go:356] "Generic (PLEG): container finished" podID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerID="ba0b1b1a3f3c8db50b08e48880c23606a996fbae12a8a12ebebc971723fc6152" exitCode=60
Nov 03 01:00:06.544736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.544668    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" event={"ID":"9e0a52de-9acc-4c3e-bb27-fa6137fee5df","Type":"ContainerDied","Data":"ba0b1b1a3f3c8db50b08e48880c23606a996fbae12a8a12ebebc971723fc6152"}
Nov 03 01:00:06.544939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.544927    4550 scope.go:122] "RemoveContainer" containerID="ba0b1b1a3f3c8db50b08e48880c23606a996fbae12a8a12ebebc971723fc6152"
Nov 03 01:00:06.545145 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.545126    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn_odh-ai-gateway-infra(9e0a52de-9acc-4c3e-bb27-fa6137fee5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df"
Nov 03 01:00:06.546833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.546811    4550 generic.go:356] "Generic (PLEG): container finished" podID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerID="70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e" exitCode=60
Nov 03 01:00:06.546932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.546888    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" event={"ID":"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c","Type":"ContainerDied","Data":"70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e"}
Nov 03 01:00:06.547174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.547157    4550 scope.go:122] "RemoveContainer" containerID="70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e"
Nov 03 01:00:06.547388 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.547366    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk_odh-ai-gateway-infra(95e9e7b8-65ef-47db-bb3e-8a62be6eef4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c"
Nov 03 01:00:06.549259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.549241    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9598928-b49d-40ab-9e70-e127b122c329" containerID="d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130" exitCode=60
Nov 03 01:00:06.549336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.549306    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerDied","Data":"d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130"}
Nov 03 01:00:06.549540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.549522    4550 scope.go:122] "RemoveContainer" containerID="d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130"
Nov 03 01:00:06.549749 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.549728    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl_odh-ai-gateway-infra(d9598928-b49d-40ab-9e70-e127b122c329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" podUID="d9598928-b49d-40ab-9e70-e127b122c329"
Nov 03 01:00:06.558609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.557724    4550 generic.go:356] "Generic (PLEG): container finished" podID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerID="9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f" exitCode=60
Nov 03 01:00:06.558609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.557802    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerDied","Data":"9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f"}
Nov 03 01:00:06.558609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.558186    4550 scope.go:122] "RemoveContainer" containerID="9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f"
Nov 03 01:00:06.558609 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.558566    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh_odh-ai-gateway-infra(34a5ddfa-213d-472a-9460-3e770c9cb1a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8"
Nov 03 01:00:06.561568 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.561542    4550 generic.go:356] "Generic (PLEG): container finished" podID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerID="fd382235f300fdbdf64bf72e9b054fc277c5232c98a007905ef37f1986247d66" exitCode=60
Nov 03 01:00:06.561690 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.561620    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" event={"ID":"fd75e99a-1821-4bb0-8f2e-16ce5c44db28","Type":"ContainerDied","Data":"fd382235f300fdbdf64bf72e9b054fc277c5232c98a007905ef37f1986247d66"}
Nov 03 01:00:06.561934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.561913    4550 scope.go:122] "RemoveContainer" containerID="fd382235f300fdbdf64bf72e9b054fc277c5232c98a007905ef37f1986247d66"
Nov 03 01:00:06.562251 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.562227    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn_odh-ai-gateway-infra(fd75e99a-1821-4bb0-8f2e-16ce5c44db28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28"
Nov 03 01:00:06.564138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.564121    4550 generic.go:356] "Generic (PLEG): container finished" podID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerID="acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306" exitCode=60
Nov 03 01:00:06.564312 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.564176    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerDied","Data":"acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306"}
Nov 03 01:00:06.564587 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.564567    4550 scope.go:122] "RemoveContainer" containerID="acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306"
Nov 03 01:00:06.564875 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.564845    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc_odh-ai-gateway-infra(068293c7-3cf2-4d37-94c7-958ca42a55f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5"
Nov 03 01:00:06.566923 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.566900    4550 generic.go:356] "Generic (PLEG): container finished" podID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerID="a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4" exitCode=60
Nov 03 01:00:06.567077 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.567062    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerDied","Data":"a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4"}
Nov 03 01:00:06.567284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:06.567264    4550 scope.go:122] "RemoveContainer" containerID="a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4"
Nov 03 01:00:06.567608 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:06.567584    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp_odh-ai-gateway-infra(e70efcea-af98-4d4a-bd08-490cb86f055b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b"
Nov 03 01:00:07.268321 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.268300    4550 scope.go:122] "RemoveContainer" containerID="c526881265ac7a6896e923588feca44dc4ecbf7f1f03930f596fcad26753fbf4"
Nov 03 01:00:07.282822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.282799    4550 scope.go:122] "RemoveContainer" containerID="17ba1e28fb49bcbff490a5c5d6d2894c691b0b2457f76b0e74dee3c2247ac1ed"
Nov 03 01:00:07.298319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.298295    4550 scope.go:122] "RemoveContainer" containerID="4b9a1ca8a34dc9a9c8c3e57a88703850ff0986992b8ba756f1436dba829ca303"
Nov 03 01:00:07.313838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.313815    4550 scope.go:122] "RemoveContainer" containerID="2cab6e0ecfb597a91f61c173f95728a1779dff1ee4ca5602c0d7e60a2b9b8cf4"
Nov 03 01:00:07.330873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.330853    4550 scope.go:122] "RemoveContainer" containerID="f5e9f03b2578288bfce9bda4bd2e1e16c277b7eb7775015b3a8c422a350e221c"
Nov 03 01:00:07.346704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.346686    4550 scope.go:122] "RemoveContainer" containerID="8cce9b93a68bdb1b70d77f2fc5b11f46d70bd3c47e327d68a7675432f6a8cfd8"
Nov 03 01:00:07.361502 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.361477    4550 scope.go:122] "RemoveContainer" containerID="5e549c46aed23e617a1d94f42f9fa77831825871d0c0107ae17910f5c3f71718"
Nov 03 01:00:07.376895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.376876    4550 scope.go:122] "RemoveContainer" containerID="dba986e1ee6092e2c7130e250ebe8d372bd3b088351a7d1c76728f8169a44094"
Nov 03 01:00:07.392849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.392823    4550 scope.go:122] "RemoveContainer" containerID="7131f0c21a531b8c10d3b7225deff88f05ec543c3f3b22c98478fc37987b9fda"
Nov 03 01:00:07.411918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.411821    4550 scope.go:122] "RemoveContainer" containerID="8f8aad49b00ea9b437f31650188b7c2096a0547f9682374d82a737fd48d0ccfc"
Nov 03 01:00:07.429393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.429372    4550 scope.go:122] "RemoveContainer" containerID="756723d3c96ec6a54b608c940d94e64a5c072cb5bb1b48bd47a91af7b23b31dd"
Nov 03 01:00:07.443938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.443919    4550 scope.go:122] "RemoveContainer" containerID="0d8ce831b3612c0194e82e403d63f2483eb4e9c7853ebe26de156bb9c723f020"
Nov 03 01:00:07.459104 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.459083    4550 scope.go:122] "RemoveContainer" containerID="6f0666d1f08805e04702ba8105445252cb5aca91bb703bafa88993a992c62923"
Nov 03 01:00:07.473498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.473481    4550 scope.go:122] "RemoveContainer" containerID="22e07ada143c0524037d4269fa5f248f097c96281203677a73d0fcc71ff58b3d"
Nov 03 01:00:07.488158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.488140    4550 scope.go:122] "RemoveContainer" containerID="a78e048631dc97261cac0dcd61296d6f3a758ecb96b26ce95b97f55d51910cc9"
Nov 03 01:00:07.504409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.504322    4550 scope.go:122] "RemoveContainer" containerID="344452d376c55529a85bdcdef0277f26b80194f4bf720c8d8e7b5eed83591ee4"
Nov 03 01:00:07.519290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.519271    4550 scope.go:122] "RemoveContainer" containerID="8b668b0f94042058e603bbd410ae6b2800b2b7721fbe411d87f7c57957151867"
Nov 03 01:00:07.535632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.535612    4550 scope.go:122] "RemoveContainer" containerID="92f2779cf857a072b50207dbc5768c2f9df59855b57fb965ce99568362394f15"
Nov 03 01:00:07.559334 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.558632    4550 scope.go:122] "RemoveContainer" containerID="9666f4f05e52f876ca137856d34838d5a7eac35ef9077deb2b4de88aeda87f79"
Nov 03 01:00:07.584052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.584025    4550 scope.go:122] "RemoveContainer" containerID="04fb01e23017d6e60104701f45cdddbbe1af5def81eafd7217dd9c9b471e75e4"
Nov 03 01:00:07.584556 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.584526    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc_odh-ai-gateway-infra(deec6453-23fd-49d0-aded-4518c0e1b5e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2"
Nov 03 01:00:07.586563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.586538    4550 scope.go:122] "RemoveContainer" containerID="ba0b1b1a3f3c8db50b08e48880c23606a996fbae12a8a12ebebc971723fc6152"
Nov 03 01:00:07.586932 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.586907    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn_odh-ai-gateway-infra(9e0a52de-9acc-4c3e-bb27-fa6137fee5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df"
Nov 03 01:00:07.589117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.589052    4550 scope.go:122] "RemoveContainer" containerID="70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e"
Nov 03 01:00:07.589529 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.589508    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk_odh-ai-gateway-infra(95e9e7b8-65ef-47db-bb3e-8a62be6eef4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c"
Nov 03 01:00:07.591076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.591056    4550 scope.go:122] "RemoveContainer" containerID="d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130"
Nov 03 01:00:07.591354 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.591328    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl_odh-ai-gateway-infra(d9598928-b49d-40ab-9e70-e127b122c329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" podUID="d9598928-b49d-40ab-9e70-e127b122c329"
Nov 03 01:00:07.592898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.592881    4550 scope.go:122] "RemoveContainer" containerID="9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f"
Nov 03 01:00:07.593171 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.593140    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh_odh-ai-gateway-infra(34a5ddfa-213d-472a-9460-3e770c9cb1a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8"
Nov 03 01:00:07.594613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.594593    4550 scope.go:122] "RemoveContainer" containerID="fd382235f300fdbdf64bf72e9b054fc277c5232c98a007905ef37f1986247d66"
Nov 03 01:00:07.594859 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.594840    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn_odh-ai-gateway-infra(fd75e99a-1821-4bb0-8f2e-16ce5c44db28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28"
Nov 03 01:00:07.596186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.596170    4550 scope.go:122] "RemoveContainer" containerID="acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306"
Nov 03 01:00:07.596409 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.596386    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc_odh-ai-gateway-infra(068293c7-3cf2-4d37-94c7-958ca42a55f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5"
Nov 03 01:00:07.597885 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.597866    4550 scope.go:122] "RemoveContainer" containerID="edabe2f374efe1b1762378d4b6a71c70bcdcd3885707be5449b230222c0d0edd"
Nov 03 01:00:07.598123 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.598098    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj_odh-ai-gateway-infra(aa394189-738e-4dc7-8cef-c0576e53a98d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d"
Nov 03 01:00:07.599675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.599654    4550 scope.go:122] "RemoveContainer" containerID="d69e30e9ca87ea4940e5e150fc9f080a2f57564bc0de49df7122b778f65d686e"
Nov 03 01:00:07.599935 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.599904    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d_odh-ai-gateway-infra(4ca04320-6331-4640-b213-c761707a2024)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" podUID="4ca04320-6331-4640-b213-c761707a2024"
Nov 03 01:00:07.601267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.601248    4550 scope.go:122] "RemoveContainer" containerID="2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5"
Nov 03 01:00:07.601569 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.601540    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-7bhb4_odh-ai-gateway-infra(c3442fb4-b070-4d02-b0a7-0f78e83f5c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61"
Nov 03 01:00:07.603040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.603023    4550 scope.go:122] "RemoveContainer" containerID="6846393c7746d6d8c8e0ffba0e22d3b695b912812a2aa9f2c60b181746b0b436"
Nov 03 01:00:07.603251 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.603228    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx_odh-ai-gateway-infra(90507494-4818-4eb4-a1cb-a3e682b3bdd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8"
Nov 03 01:00:07.604720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.604703    4550 scope.go:122] "RemoveContainer" containerID="6d51ee38a9656a9b5dd7623b5f150a0537bb091a81be377d0aa99e949daf5b23"
Nov 03 01:00:07.604934 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.604912    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh_odh-ai-gateway-infra(36a4e5e2-d23b-4006-a3fa-448ded9b054f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f"
Nov 03 01:00:07.606731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.606704    4550 generic.go:356] "Generic (PLEG): container finished" podID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerID="39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60" exitCode=60
Nov 03 01:00:07.606847 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.606795    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerDied","Data":"39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60"}
Nov 03 01:00:07.606911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.606876    4550 scope.go:122] "RemoveContainer" containerID="59ef4517d9bdefedf8c6f1961767912d79346f8a1bdc83954f1d1de1b3263919"
Nov 03 01:00:07.607242 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.607222    4550 scope.go:122] "RemoveContainer" containerID="39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60"
Nov 03 01:00:07.607524 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.607499    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln_odh-ai-gateway-infra(f53c2794-70d3-497e-91f6-08e4ec16bf59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59"
Nov 03 01:00:07.609205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.609187    4550 scope.go:122] "RemoveContainer" containerID="d6b94d1d651340b907b155805006034ebf3bbdf46f166d718203b8f38b416035"
Nov 03 01:00:07.609393 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.609376    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc_odh-ai-gateway-infra(03449032-8806-4415-a9ff-7ac5f37df4ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab"
Nov 03 01:00:07.611090 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.611065    4550 generic.go:356] "Generic (PLEG): container finished" podID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerID="1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849" exitCode=60
Nov 03 01:00:07.611175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.611134    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerDied","Data":"1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849"}
Nov 03 01:00:07.611324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.611306    4550 scope.go:122] "RemoveContainer" containerID="1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849"
Nov 03 01:00:07.611552 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.611534    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x_odh-ai-gateway-infra(1cd14ecf-d850-4d11-9951-22c62bea5ffb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb"
Nov 03 01:00:07.613167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.613150    4550 scope.go:122] "RemoveContainer" containerID="61ccc9a8e8f854a5d31cd113b5f33ec1fa36575a4e2aa13fed94c124221609fb"
Nov 03 01:00:07.613366 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.613348    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c_odh-ai-gateway-infra(30c9afba-79f5-41d3-ae1e-0cf85863e6d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6"
Nov 03 01:00:07.614974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.614951    4550 scope.go:122] "RemoveContainer" containerID="790acc2e378b37299c695f0fc3e4bb7974abc5a770fd1def0aca63de923dd655"
Nov 03 01:00:07.615241 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.615218    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx_odh-ai-gateway-infra(4171a7bd-72e5-4863-9cdb-5637237dcddf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf"
Nov 03 01:00:07.616553 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.616533    4550 scope.go:122] "RemoveContainer" containerID="52a2c6db53b3e99542319888946cd4af68b467b949aaad70fb625875c30433fb"
Nov 03 01:00:07.616817 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.616795    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6_odh-ai-gateway-infra(1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce"
Nov 03 01:00:07.618319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.618301    4550 scope.go:122] "RemoveContainer" containerID="c19d4a9fee7405c1a5249f75a192ed58598858438402938a87b7f87b8b5ff634"
Nov 03 01:00:07.618578 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.618557    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp_odh-ai-gateway-infra(620eee4a-a44a-45ce-8cdc-b9a4368aa6ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab"
Nov 03 01:00:07.620306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.620285    4550 generic.go:356] "Generic (PLEG): container finished" podID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerID="8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad" exitCode=60
Nov 03 01:00:07.620404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.620329    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerDied","Data":"8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad"}
Nov 03 01:00:07.620622 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.620607    4550 scope.go:122] "RemoveContainer" containerID="8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad"
Nov 03 01:00:07.620881 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.620857    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j_odh-ai-gateway-infra(540463f3-b44b-4da2-adeb-192f817fcf3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c"
Nov 03 01:00:07.622848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.622831    4550 scope.go:122] "RemoveContainer" containerID="0b3e2f751cf36128879a8a713b459fbe1637a95d7834682268a2347cad8a3c43"
Nov 03 01:00:07.623096 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.623075    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n_odh-ai-gateway-infra(8f332463-51f7-4d57-ae51-99f46e0074b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6"
Nov 03 01:00:07.630264 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.630241    4550 scope.go:122] "RemoveContainer" containerID="1db6ad126af8d25bf3776c215b38446b8f4d2db3453c42e6e89baf1beb8dcc79"
Nov 03 01:00:07.634322 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.634297    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:07.635311 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.634710    4550 scope.go:122] "RemoveContainer" containerID="a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4"
Nov 03 01:00:07.635476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.635454    4550 scope.go:122] "RemoveContainer" containerID="2a057aea2252a251ca4f11b8b3a70f09921304d3e3b47c54d0abd9d19bab6f4a"
Nov 03 01:00:07.635739 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.635717    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp_odh-ai-gateway-infra(e70efcea-af98-4d4a-bd08-490cb86f055b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b"
Nov 03 01:00:07.636002 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:07.635975    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b_odh-ai-gateway-infra(3117c03f-2cc4-467a-8716-fb4e1b6c0f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e"
Nov 03 01:00:07.658637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:07.658615    4550 scope.go:122] "RemoveContainer" containerID="914a7cc3951a9f37fe8c4b4d88381873cf5f1ccfd4599cd9234f49d36410a1ec"
Nov 03 01:00:08.647140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:08.647096    4550 scope.go:122] "RemoveContainer" containerID="39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60"
Nov 03 01:00:08.648046 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:08.648016    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln_odh-ai-gateway-infra(f53c2794-70d3-497e-91f6-08e4ec16bf59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59"
Nov 03 01:00:08.648967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:08.648934    4550 scope.go:122] "RemoveContainer" containerID="1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849"
Nov 03 01:00:08.649369 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:08.649334    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x_odh-ai-gateway-infra(1cd14ecf-d850-4d11-9951-22c62bea5ffb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb"
Nov 03 01:00:08.650502 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:08.650480    4550 scope.go:122] "RemoveContainer" containerID="8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad"
Nov 03 01:00:08.650795 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:08.650777    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j_odh-ai-gateway-infra(540463f3-b44b-4da2-adeb-192f817fcf3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c"
Nov 03 01:00:09.313155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:09.313124    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:11.038485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:11.038413    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:12.742992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:12.742954    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:14.090927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:14.090894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:14.189375 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:14.189339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:14.450842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:14.450811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:16.135899 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:16.135869    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:16.800834 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:16.800801    4550 scope.go:122] "RemoveContainer" containerID="fd382235f300fdbdf64bf72e9b054fc277c5232c98a007905ef37f1986247d66"
Nov 03 01:00:16.801083 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:16.801065    4550 scope.go:122] "RemoveContainer" containerID="2a057aea2252a251ca4f11b8b3a70f09921304d3e3b47c54d0abd9d19bab6f4a"
Nov 03 01:00:17.747268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.747236    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerStarted","Data":"e9571216614d03c530f3b08f0b802fa4dc1b3cab1e9bed53eb74a9c519d400ae"}
Nov 03 01:00:17.749581 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.749555    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" event={"ID":"fd75e99a-1821-4bb0-8f2e-16ce5c44db28","Type":"ContainerStarted","Data":"d64adcd180803d706a4a410016759b875d1729f9bfebd66d5a2885ac9a8b1019"}
Nov 03 01:00:17.801336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.801297    4550 scope.go:122] "RemoveContainer" containerID="52a2c6db53b3e99542319888946cd4af68b467b949aaad70fb625875c30433fb"
Nov 03 01:00:17.801538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.801468    4550 scope.go:122] "RemoveContainer" containerID="6846393c7746d6d8c8e0ffba0e22d3b695b912812a2aa9f2c60b181746b0b436"
Nov 03 01:00:17.801681 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.801659    4550 scope.go:122] "RemoveContainer" containerID="1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849"
Nov 03 01:00:17.801799 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.801758    4550 scope.go:122] "RemoveContainer" containerID="61ccc9a8e8f854a5d31cd113b5f33ec1fa36575a4e2aa13fed94c124221609fb"
Nov 03 01:00:17.831225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.831172    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" podStartSLOduration=17.83115268 podStartE2EDuration="17.83115268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:17.824821552 +0000 UTC m=+3300.146358363" watchObservedRunningTime="2026-11-03 01:00:17.83115268 +0000 UTC m=+3300.152689480"
Nov 03 01:00:17.935907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:17.935882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:18.770937 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.770851    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerStarted","Data":"ed6a5011361d47c6c2f022c9d21a40d58c68555bf9df875a78ea4e39345715bb"}
Nov 03 01:00:18.773459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.773397    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerStarted","Data":"d17f7078e904501b3f1a96bc6e765c0e994d05cb7b1cace9fca14ea4c065ee74"}
Nov 03 01:00:18.775471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.775451    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerStarted","Data":"d5ad885014533b398b15d27aa26e4146d21f11a954ba0f159cd1f9fa0b0c954c"}
Nov 03 01:00:18.777750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.777726    4550 generic.go:356] "Generic (PLEG): container finished" podID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerID="e9571216614d03c530f3b08f0b802fa4dc1b3cab1e9bed53eb74a9c519d400ae" exitCode=60
Nov 03 01:00:18.777844 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.777784    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerDied","Data":"e9571216614d03c530f3b08f0b802fa4dc1b3cab1e9bed53eb74a9c519d400ae"}
Nov 03 01:00:18.777844 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.777829    4550 scope.go:122] "RemoveContainer" containerID="2a057aea2252a251ca4f11b8b3a70f09921304d3e3b47c54d0abd9d19bab6f4a"
Nov 03 01:00:18.778154 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.778136    4550 scope.go:122] "RemoveContainer" containerID="e9571216614d03c530f3b08f0b802fa4dc1b3cab1e9bed53eb74a9c519d400ae"
Nov 03 01:00:18.778587 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:18.778549    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b_odh-ai-gateway-infra(3117c03f-2cc4-467a-8716-fb4e1b6c0f5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e"
Nov 03 01:00:18.779979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.779957    4550 generic.go:356] "Generic (PLEG): container finished" podID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerID="d64adcd180803d706a4a410016759b875d1729f9bfebd66d5a2885ac9a8b1019" exitCode=60
Nov 03 01:00:18.780067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.780025    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" event={"ID":"fd75e99a-1821-4bb0-8f2e-16ce5c44db28","Type":"ContainerDied","Data":"d64adcd180803d706a4a410016759b875d1729f9bfebd66d5a2885ac9a8b1019"}
Nov 03 01:00:18.780345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.780329    4550 scope.go:122] "RemoveContainer" containerID="d64adcd180803d706a4a410016759b875d1729f9bfebd66d5a2885ac9a8b1019"
Nov 03 01:00:18.780636 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:18.780614    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn_odh-ai-gateway-infra(fd75e99a-1821-4bb0-8f2e-16ce5c44db28)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28"
Nov 03 01:00:18.782189 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.782170    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" event={"ID":"90507494-4818-4eb4-a1cb-a3e682b3bdd8","Type":"ContainerStarted","Data":"d03ee6c549e370f78360e164c4578db17a4024df1ce9052420df353dd4dee823"}
Nov 03 01:00:18.799735 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.799602    4550 scope.go:122] "RemoveContainer" containerID="fd382235f300fdbdf64bf72e9b054fc277c5232c98a007905ef37f1986247d66"
Nov 03 01:00:18.801766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.801739    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"]
Nov 03 01:00:18.802309 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.802286    4550 scope.go:122] "RemoveContainer" containerID="790acc2e378b37299c695f0fc3e4bb7974abc5a770fd1def0aca63de923dd655"
Nov 03 01:00:18.803750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.803730    4550 scope.go:122] "RemoveContainer" containerID="39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60"
Nov 03 01:00:18.804470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.804410    4550 scope.go:122] "RemoveContainer" containerID="ba0b1b1a3f3c8db50b08e48880c23606a996fbae12a8a12ebebc971723fc6152"
Nov 03 01:00:18.804895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.804873    4550 scope.go:122] "RemoveContainer" containerID="acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306"
Nov 03 01:00:18.806784 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.806761    4550 scope.go:122] "RemoveContainer" containerID="6d51ee38a9656a9b5dd7623b5f150a0537bb091a81be377d0aa99e949daf5b23"
Nov 03 01:00:18.830311 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.830104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:18.844534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.844515    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"]
Nov 03 01:00:18.874374 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.874326    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" podStartSLOduration=18.874311573 podStartE2EDuration="18.874311573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:18.863926518 +0000 UTC m=+3301.185463319" watchObservedRunningTime="2026-11-03 01:00:18.874311573 +0000 UTC m=+3301.195848372"
Nov 03 01:00:18.930917 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:18.930866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:19.670237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.670107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:19.827654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.827625    4550 generic.go:356] "Generic (PLEG): container finished" podID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerID="d03ee6c549e370f78360e164c4578db17a4024df1ce9052420df353dd4dee823" exitCode=60
Nov 03 01:00:19.831815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.831772    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" event={"ID":"90507494-4818-4eb4-a1cb-a3e682b3bdd8","Type":"ContainerDied","Data":"d03ee6c549e370f78360e164c4578db17a4024df1ce9052420df353dd4dee823"}
Nov 03 01:00:19.832003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.831823    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"]
Nov 03 01:00:19.832226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.832199    4550 scope.go:122] "RemoveContainer" containerID="6846393c7746d6d8c8e0ffba0e22d3b695b912812a2aa9f2c60b181746b0b436"
Nov 03 01:00:19.832811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.832779    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerStarted","Data":"0c0f0cb3b90f356ce6573fa3cc5f56a654ab39e9cebba190f3896b6ddeaa3672"}
Nov 03 01:00:19.834906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.834886    4550 scope.go:122] "RemoveContainer" containerID="70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e"
Nov 03 01:00:19.835022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.834903    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerDied","Data":"ed6a5011361d47c6c2f022c9d21a40d58c68555bf9df875a78ea4e39345715bb"}
Nov 03 01:00:19.835096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.835057    4550 scope.go:122] "RemoveContainer" containerID="c19d4a9fee7405c1a5249f75a192ed58598858438402938a87b7f87b8b5ff634"
Nov 03 01:00:19.835307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.835288    4550 scope.go:122] "RemoveContainer" containerID="9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f"
Nov 03 01:00:19.835394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.834885    4550 generic.go:356] "Generic (PLEG): container finished" podID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerID="ed6a5011361d47c6c2f022c9d21a40d58c68555bf9df875a78ea4e39345715bb" exitCode=60
Nov 03 01:00:19.835602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.835581    4550 scope.go:122] "RemoveContainer" containerID="d03ee6c549e370f78360e164c4578db17a4024df1ce9052420df353dd4dee823"
Nov 03 01:00:19.835653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.835623    4550 scope.go:122] "RemoveContainer" containerID="d6b94d1d651340b907b155805006034ebf3bbdf46f166d718203b8f38b416035"
Nov 03 01:00:19.836418 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:19.835902    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx_odh-ai-gateway-infra(90507494-4818-4eb4-a1cb-a3e682b3bdd8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8"
Nov 03 01:00:19.836418 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.836142    4550 scope.go:122] "RemoveContainer" containerID="04fb01e23017d6e60104701f45cdddbbe1af5def81eafd7217dd9c9b471e75e4"
Nov 03 01:00:19.836876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.836656    4550 scope.go:122] "RemoveContainer" containerID="edabe2f374efe1b1762378d4b6a71c70bcdcd3885707be5449b230222c0d0edd"
Nov 03 01:00:19.837031 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.836879    4550 scope.go:122] "RemoveContainer" containerID="d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130"
Nov 03 01:00:19.840125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.837989    4550 scope.go:122] "RemoveContainer" containerID="a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4"
Nov 03 01:00:19.843501 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.843251    4550 generic.go:356] "Generic (PLEG): container finished" podID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerID="d17f7078e904501b3f1a96bc6e765c0e994d05cb7b1cace9fca14ea4c065ee74" exitCode=60
Nov 03 01:00:19.843501 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.843368    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerDied","Data":"d17f7078e904501b3f1a96bc6e765c0e994d05cb7b1cace9fca14ea4c065ee74"}
Nov 03 01:00:19.844084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.843792    4550 scope.go:122] "RemoveContainer" containerID="d17f7078e904501b3f1a96bc6e765c0e994d05cb7b1cace9fca14ea4c065ee74"
Nov 03 01:00:19.844638 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:19.844557    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c_odh-ai-gateway-infra(30c9afba-79f5-41d3-ae1e-0cf85863e6d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6"
Nov 03 01:00:19.848079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.848060    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" event={"ID":"4171a7bd-72e5-4863-9cdb-5637237dcddf","Type":"ContainerStarted","Data":"3de797920c4cf25d68828a35aaa5e0ac17edecadf9663f862fbb66cbd236ef68"}
Nov 03 01:00:19.850266 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.850247    4550 generic.go:356] "Generic (PLEG): container finished" podID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerID="d5ad885014533b398b15d27aa26e4146d21f11a954ba0f159cd1f9fa0b0c954c" exitCode=60
Nov 03 01:00:19.850384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.850348    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerDied","Data":"d5ad885014533b398b15d27aa26e4146d21f11a954ba0f159cd1f9fa0b0c954c"}
Nov 03 01:00:19.850635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.850620    4550 scope.go:122] "RemoveContainer" containerID="d5ad885014533b398b15d27aa26e4146d21f11a954ba0f159cd1f9fa0b0c954c"
Nov 03 01:00:19.850856 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:19.850836    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6_odh-ai-gateway-infra(1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce"
Nov 03 01:00:19.875493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.873089    4550 scope.go:122] "RemoveContainer" containerID="1a9327d80d7a43344bf5679b882a6ebb80d52fc3a77293149d2ae9d98fd72849"
Nov 03 01:00:19.877197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.877172    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"]
Nov 03 01:00:19.900883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.900783    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerStarted","Data":"ecd8f0f04d31357a35088d142ff3888920bdd31ddb094392a68651524faf37b9"}
Nov 03 01:00:19.933604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.933543    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"]
Nov 03 01:00:19.998704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:19.998656    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"]
Nov 03 01:00:20.238385 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.238342    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" podStartSLOduration=20.238321529 podStartE2EDuration="20.238321529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:20.237721247 +0000 UTC m=+3302.559258049" watchObservedRunningTime="2026-11-03 01:00:20.238321529 +0000 UTC m=+3302.559858365"
Nov 03 01:00:20.338527 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.338486    4550 scope.go:122] "RemoveContainer" containerID="61ccc9a8e8f854a5d31cd113b5f33ec1fa36575a4e2aa13fed94c124221609fb"
Nov 03 01:00:20.345047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.345025    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:20.357540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.357513    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:20.359456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.359422    4550 scope.go:122] "RemoveContainer" containerID="52a2c6db53b3e99542319888946cd4af68b467b949aaad70fb625875c30433fb"
Nov 03 01:00:20.404662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.403389    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca\") pod \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") "
Nov 03 01:00:20.404662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.403451    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr\" (UniqueName: \"kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr\") pod \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") "
Nov 03 01:00:20.404662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.403606    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca\") pod \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\" (UID: \"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e\") "
Nov 03 01:00:20.404662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.403637    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl\" (UniqueName: \"kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl\") pod \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\" (UID: \"1cd14ecf-d850-4d11-9951-22c62bea5ffb\") "
Nov 03 01:00:20.404662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.403861    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca" pod "1cd14ecf-d850-4d11-9951-22c62bea5ffb" (UID: "1cd14ecf-d850-4d11-9951-22c62bea5ffb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:20.404662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.404360    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd14ecf-d850-4d11-9951-22c62bea5ffb-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.405335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.405310    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca" pod "3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" (UID: "3117c03f-2cc4-467a-8716-fb4e1b6c0f5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:20.407069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.407048    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl" pod "1cd14ecf-d850-4d11-9951-22c62bea5ffb" (UID: "1cd14ecf-d850-4d11-9951-22c62bea5ffb"). InnerVolumeSpecName "kube-api-access-b2wzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:20.407319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.407294    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr" pod "3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" (UID: "3117c03f-2cc4-467a-8716-fb4e1b6c0f5e"). InnerVolumeSpecName "kube-api-access-dg4mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:20.443871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.443847    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:20.505254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.505175    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z\" (UniqueName: \"kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z\") pod \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") "
Nov 03 01:00:20.505774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.505416    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca\") pod \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\" (UID: \"fd75e99a-1821-4bb0-8f2e-16ce5c44db28\") "
Nov 03 01:00:20.505905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.505742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca" pod "fd75e99a-1821-4bb0-8f2e-16ce5c44db28" (UID: "fd75e99a-1821-4bb0-8f2e-16ce5c44db28"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:20.506326 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.506304    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg4mr\" (UniqueName: \"kubernetes.io/projected/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-kube-api-access-dg4mr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.506403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.506359    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.506403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.506371    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.506403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.506381    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2wzl\" (UniqueName: \"kubernetes.io/projected/1cd14ecf-d850-4d11-9951-22c62bea5ffb-kube-api-access-b2wzl\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.580878 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.580853    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z" pod "fd75e99a-1821-4bb0-8f2e-16ce5c44db28" (UID: "fd75e99a-1821-4bb0-8f2e-16ce5c44db28"). InnerVolumeSpecName "kube-api-access-9xp5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:20.607658 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.607634    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xp5z\" (UniqueName: \"kubernetes.io/projected/fd75e99a-1821-4bb0-8f2e-16ce5c44db28-kube-api-access-9xp5z\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.785311 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.785284    4550 scope.go:122] "RemoveContainer" containerID="528431011aeaa00b806ff488bdfd3e9b00c65c60be3f5bea4c9f9afc12bec9ef"
Nov 03 01:00:20.800134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.800113    4550 scope.go:122] "RemoveContainer" containerID="872daa101ecc38988c5f3659fb63ba1a30128e7ea34de78bc96a90c58d30a106"
Nov 03 01:00:20.800907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.800878    4550 scope.go:122] "RemoveContainer" containerID="d69e30e9ca87ea4940e5e150fc9f080a2f57564bc0de49df7122b778f65d686e"
Nov 03 01:00:20.801117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.801095    4550 scope.go:122] "RemoveContainer" containerID="2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5"
Nov 03 01:00:20.801348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.801320    4550 scope.go:122] "RemoveContainer" containerID="0b3e2f751cf36128879a8a713b459fbe1637a95d7834682268a2347cad8a3c43"
Nov 03 01:00:20.801515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.801449    4550 scope.go:122] "RemoveContainer" containerID="8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad"
Nov 03 01:00:20.911785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.911760    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" event={"ID":"aa394189-738e-4dc7-8cef-c0576e53a98d","Type":"ContainerStarted","Data":"0a6f1e1f6d2003c7ec4dd694c047b7fb5091c12beb33e43a537d344eadd36c35"}
Nov 03 01:00:20.916549 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.916527    4550 generic.go:356] "Generic (PLEG): container finished" podID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerID="b1e78289e9d4a65c2e73f09901acce79c49ebfb8acc62bb846aa9c28865ecd39" exitCode=60
Nov 03 01:00:20.916665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.916643    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" event={"ID":"36a4e5e2-d23b-4006-a3fa-448ded9b054f","Type":"ContainerDied","Data":"b1e78289e9d4a65c2e73f09901acce79c49ebfb8acc62bb846aa9c28865ecd39"}
Nov 03 01:00:20.916722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.916687    4550 scope.go:122] "RemoveContainer" containerID="6d51ee38a9656a9b5dd7623b5f150a0537bb091a81be377d0aa99e949daf5b23"
Nov 03 01:00:20.916993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.916979    4550 scope.go:122] "RemoveContainer" containerID="b1e78289e9d4a65c2e73f09901acce79c49ebfb8acc62bb846aa9c28865ecd39"
Nov 03 01:00:20.917319 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:20.917243    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh_odh-ai-gateway-infra(36a4e5e2-d23b-4006-a3fa-448ded9b054f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f"
Nov 03 01:00:20.919136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.919117    4550 generic.go:356] "Generic (PLEG): container finished" podID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerID="0c0f0cb3b90f356ce6573fa3cc5f56a654ab39e9cebba190f3896b6ddeaa3672" exitCode=60
Nov 03 01:00:20.919260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.919186    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerDied","Data":"0c0f0cb3b90f356ce6573fa3cc5f56a654ab39e9cebba190f3896b6ddeaa3672"}
Nov 03 01:00:20.923292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.923262    4550 scope.go:122] "RemoveContainer" containerID="0c0f0cb3b90f356ce6573fa3cc5f56a654ab39e9cebba190f3896b6ddeaa3672"
Nov 03 01:00:20.923975 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:20.923941    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln_odh-ai-gateway-infra(f53c2794-70d3-497e-91f6-08e4ec16bf59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59"
Nov 03 01:00:20.929756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.929731    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x" event={"ID":"1cd14ecf-d850-4d11-9951-22c62bea5ffb","Type":"ContainerDied","Data":"3eed94e313393a7ace836ba4554c1965536f32bf643d60eb339d9f3a7ffd67b1"}
Nov 03 01:00:20.930805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.929893    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"
Nov 03 01:00:20.935716 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.935665    4550 generic.go:356] "Generic (PLEG): container finished" podID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerID="3de797920c4cf25d68828a35aaa5e0ac17edecadf9663f862fbb66cbd236ef68" exitCode=60
Nov 03 01:00:20.936959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.936937    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" event={"ID":"4171a7bd-72e5-4863-9cdb-5637237dcddf","Type":"ContainerDied","Data":"3de797920c4cf25d68828a35aaa5e0ac17edecadf9663f862fbb66cbd236ef68"}
Nov 03 01:00:20.937569 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.937549    4550 scope.go:122] "RemoveContainer" containerID="3de797920c4cf25d68828a35aaa5e0ac17edecadf9663f862fbb66cbd236ef68"
Nov 03 01:00:20.938273 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:20.938249    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx_odh-ai-gateway-infra(4171a7bd-72e5-4863-9cdb-5637237dcddf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf"
Nov 03 01:00:20.946382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.946240    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" event={"ID":"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab","Type":"ContainerStarted","Data":"1bd586f63182737faf68a49920c71bbef63d81cfc38c83c8f4fe9898fb8518fd"}
Nov 03 01:00:20.971707 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.971681    4550 scope.go:122] "RemoveContainer" containerID="39915b4cc4750d221828eedd950d5f17692a3e8be0aee869e56fd559b345ea60"
Nov 03 01:00:20.988071 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.982896    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" podStartSLOduration=20.982878093 podStartE2EDuration="20.982878093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:20.955618055 +0000 UTC m=+3303.277154858" watchObservedRunningTime="2026-11-03 01:00:20.982878093 +0000 UTC m=+3303.304414893"
Nov 03 01:00:20.989251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.988620    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b" event={"ID":"3117c03f-2cc4-467a-8716-fb4e1b6c0f5e","Type":"ContainerDied","Data":"1e6c55ec5c3a36d3706b7b7b0f2abe2dedd85011c6e54ad8f53829f5bf39c72e"}
Nov 03 01:00:20.989251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:20.988764    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"
Nov 03 01:00:21.001469 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.000826    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerStarted","Data":"f463314bbcebf34698adab87240311464e442dcda902c8c7d24ccf68573ecec0"}
Nov 03 01:00:21.007484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.004376    4550 generic.go:356] "Generic (PLEG): container finished" podID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerID="c4811f98eb3f8aa79dae8d2413f70dc62d146aea7a1eca5a23f3191384fa40fb" exitCode=60
Nov 03 01:00:21.007484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.004505    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" event={"ID":"9e0a52de-9acc-4c3e-bb27-fa6137fee5df","Type":"ContainerDied","Data":"c4811f98eb3f8aa79dae8d2413f70dc62d146aea7a1eca5a23f3191384fa40fb"}
Nov 03 01:00:21.007862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.007839    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerStarted","Data":"07e62286cb7f2c147a81af4de38321e0d4fe48dcd9bb301e6859514c0ad1bb0d"}
Nov 03 01:00:21.008675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.008657    4550 scope.go:122] "RemoveContainer" containerID="c4811f98eb3f8aa79dae8d2413f70dc62d146aea7a1eca5a23f3191384fa40fb"
Nov 03 01:00:21.027060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.026935    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" podStartSLOduration=21.026916919 podStartE2EDuration="21.026916919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:21.021932994 +0000 UTC m=+3303.343469795" watchObservedRunningTime="2026-11-03 01:00:21.026916919 +0000 UTC m=+3303.348453721"
Nov 03 01:00:21.030603 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:21.030570    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn_odh-ai-gateway-infra(9e0a52de-9acc-4c3e-bb27-fa6137fee5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df"
Nov 03 01:00:21.034825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.034797    4550 scope.go:122] "RemoveContainer" containerID="ed6a5011361d47c6c2f022c9d21a40d58c68555bf9df875a78ea4e39345715bb"
Nov 03 01:00:21.037014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.036986    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerStarted","Data":"4562587d91907605d67c8ce7ee4ae02defafbc73161d5f3c9221faa41fe64421"}
Nov 03 01:00:21.050469 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.049317    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn" event={"ID":"fd75e99a-1821-4bb0-8f2e-16ce5c44db28","Type":"ContainerDied","Data":"aa77f74058e569f7505f0a4d30a314504fec3d09349315ffa31bbe842b46dfe5"}
Nov 03 01:00:21.050469 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.049468    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"
Nov 03 01:00:21.076201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.076177    4550 scope.go:122] "RemoveContainer" containerID="790acc2e378b37299c695f0fc3e4bb7974abc5a770fd1def0aca63de923dd655"
Nov 03 01:00:21.080971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.080846    4550 generic.go:356] "Generic (PLEG): container finished" podID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerID="ecd8f0f04d31357a35088d142ff3888920bdd31ddb094392a68651524faf37b9" exitCode=60
Nov 03 01:00:21.081105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.080870    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerDied","Data":"ecd8f0f04d31357a35088d142ff3888920bdd31ddb094392a68651524faf37b9"}
Nov 03 01:00:21.083369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.083347    4550 scope.go:122] "RemoveContainer" containerID="ecd8f0f04d31357a35088d142ff3888920bdd31ddb094392a68651524faf37b9"
Nov 03 01:00:21.084054 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:21.084028    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc_odh-ai-gateway-infra(068293c7-3cf2-4d37-94c7-958ca42a55f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5"
Nov 03 01:00:21.091752 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.091731    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerStarted","Data":"e534aa4299646cb533240cac379b599207aa61fb6b0aee9a5c49b7c9d03ef655"}
Nov 03 01:00:21.105025 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.104960    4550 scope.go:122] "RemoveContainer" containerID="e9571216614d03c530f3b08f0b802fa4dc1b3cab1e9bed53eb74a9c519d400ae"
Nov 03 01:00:21.121123 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.121091    4550 scope.go:122] "RemoveContainer" containerID="ba0b1b1a3f3c8db50b08e48880c23606a996fbae12a8a12ebebc971723fc6152"
Nov 03 01:00:21.140032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.140012    4550 scope.go:122] "RemoveContainer" containerID="d64adcd180803d706a4a410016759b875d1729f9bfebd66d5a2885ac9a8b1019"
Nov 03 01:00:21.163560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.159133    4550 scope.go:122] "RemoveContainer" containerID="acbbab00156e08c4569a8bc640324b2ff345df018f009ead41287ef2ac386306"
Nov 03 01:00:21.218680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.218655    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"]
Nov 03 01:00:21.250155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.250124    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"]
Nov 03 01:00:21.312474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.312405    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"]
Nov 03 01:00:21.411835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.411809    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:21.416636 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.416614    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:21.421003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.420983    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:21.454332 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.454302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:21.489246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489221    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca\") pod \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") "
Nov 03 01:00:21.489376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489257    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s\" (UniqueName: \"kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s\") pod \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") "
Nov 03 01:00:21.489513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489398    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca\") pod \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\" (UID: \"90507494-4818-4eb4-a1cb-a3e682b3bdd8\") "
Nov 03 01:00:21.489574 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489557    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb\" (UniqueName: \"kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb\") pod \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\" (UID: \"30c9afba-79f5-41d3-ae1e-0cf85863e6d6\") "
Nov 03 01:00:21.489725 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489708    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6\" (UniqueName: \"kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6\") pod \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") "
Nov 03 01:00:21.489821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489557    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca" pod "30c9afba-79f5-41d3-ae1e-0cf85863e6d6" (UID: "30c9afba-79f5-41d3-ae1e-0cf85863e6d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:21.489821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489813    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca\") pod \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\" (UID: \"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce\") "
Nov 03 01:00:21.490093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.489794    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca" pod "90507494-4818-4eb4-a1cb-a3e682b3bdd8" (UID: "90507494-4818-4eb4-a1cb-a3e682b3bdd8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:21.490162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.490134    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca" pod "1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" (UID: "1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:21.490958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.490934    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.491064 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.490962    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.491064 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.490976    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90507494-4818-4eb4-a1cb-a3e682b3bdd8-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.571705 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.570238    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb" pod "30c9afba-79f5-41d3-ae1e-0cf85863e6d6" (UID: "30c9afba-79f5-41d3-ae1e-0cf85863e6d6"). InnerVolumeSpecName "kube-api-access-mt5mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:21.571705 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.571347    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6" pod "1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" (UID: "1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce"). InnerVolumeSpecName "kube-api-access-rl9h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:21.577941 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.575794    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s" pod "90507494-4818-4eb4-a1cb-a3e682b3bdd8" (UID: "90507494-4818-4eb4-a1cb-a3e682b3bdd8"). InnerVolumeSpecName "kube-api-access-qw78s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:21.579047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.578985    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"]
Nov 03 01:00:21.590206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.590178    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-065de0-29894460-gkn4b"]
Nov 03 01:00:21.599131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.599100    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt5mb\" (UniqueName: \"kubernetes.io/projected/30c9afba-79f5-41d3-ae1e-0cf85863e6d6-kube-api-access-mt5mb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.599272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.599133    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl9h6\" (UniqueName: \"kubernetes.io/projected/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce-kube-api-access-rl9h6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.599272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.599153    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw78s\" (UniqueName: \"kubernetes.io/projected/90507494-4818-4eb4-a1cb-a3e682b3bdd8-kube-api-access-qw78s\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.603181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.603079    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"]
Nov 03 01:00:21.609656 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.609632    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-e62935-29894460-d82vn"]
Nov 03 01:00:21.639164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.639143    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"]
Nov 03 01:00:21.644345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.644324    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20109d-29894460-tm72x"]
Nov 03 01:00:21.818774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.818698    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" path="/var/lib/kubelet/pods/1cd14ecf-d850-4d11-9951-22c62bea5ffb/volumes"
Nov 03 01:00:21.819738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.819713    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" path="/var/lib/kubelet/pods/3117c03f-2cc4-467a-8716-fb4e1b6c0f5e/volumes"
Nov 03 01:00:21.821014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.820611    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" path="/var/lib/kubelet/pods/fd75e99a-1821-4bb0-8f2e-16ce5c44db28/volumes"
Nov 03 01:00:21.972216 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:21.972183    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"]
Nov 03 01:00:22.030056 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.030025    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"]
Nov 03 01:00:22.108195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.108166    4550 generic.go:356] "Generic (PLEG): container finished" podID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerID="1ad86fed92b0bbfba8afa5b162c152ee8e5d6f1a9060badd473401dbdbf98c5f" exitCode=60
Nov 03 01:00:22.108336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.108234    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" event={"ID":"03449032-8806-4415-a9ff-7ac5f37df4ab","Type":"ContainerDied","Data":"1ad86fed92b0bbfba8afa5b162c152ee8e5d6f1a9060badd473401dbdbf98c5f"}
Nov 03 01:00:22.108391 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.108338    4550 scope.go:122] "RemoveContainer" containerID="d6b94d1d651340b907b155805006034ebf3bbdf46f166d718203b8f38b416035"
Nov 03 01:00:22.108689 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.108672    4550 scope.go:122] "RemoveContainer" containerID="1ad86fed92b0bbfba8afa5b162c152ee8e5d6f1a9060badd473401dbdbf98c5f"
Nov 03 01:00:22.109002 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:22.108977    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc_odh-ai-gateway-infra(03449032-8806-4415-a9ff-7ac5f37df4ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab"
Nov 03 01:00:22.110918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.110897    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c" event={"ID":"30c9afba-79f5-41d3-ae1e-0cf85863e6d6","Type":"ContainerDied","Data":"ceaf02e82ea903f0e7d30e6df562ea9e69eca4dd3d0a2c912e37de78a5f71559"}
Nov 03 01:00:22.111012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.110914    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"
Nov 03 01:00:22.114297 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.114274    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6" event={"ID":"1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce","Type":"ContainerDied","Data":"7c4f82b713810fea7cc051aa9539734d2d1828d38bb06b81a3c0309221731758"}
Nov 03 01:00:22.114394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.114281    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"
Nov 03 01:00:22.116504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.116483    4550 generic.go:356] "Generic (PLEG): container finished" podID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerID="1bd586f63182737faf68a49920c71bbef63d81cfc38c83c8f4fe9898fb8518fd" exitCode=60
Nov 03 01:00:22.116620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.116557    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" event={"ID":"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab","Type":"ContainerDied","Data":"1bd586f63182737faf68a49920c71bbef63d81cfc38c83c8f4fe9898fb8518fd"}
Nov 03 01:00:22.120233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.120207    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerStarted","Data":"26baf6a9f42af3687c062a03d2f2dea19d0e58aebc9d56974a55510cab62e142"}
Nov 03 01:00:22.123008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.122738    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" event={"ID":"8f332463-51f7-4d57-ae51-99f46e0074b6","Type":"ContainerStarted","Data":"143de626a421cb0e5d7d476bfdacdc4af5b741859f946df16c14456c11272d6d"}
Nov 03 01:00:22.126590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.126567    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"]
Nov 03 01:00:22.127607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.127536    4550 generic.go:356] "Generic (PLEG): container finished" podID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerID="f463314bbcebf34698adab87240311464e442dcda902c8c7d24ccf68573ecec0" exitCode=60
Nov 03 01:00:22.127607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.127573    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerDied","Data":"f463314bbcebf34698adab87240311464e442dcda902c8c7d24ccf68573ecec0"}
Nov 03 01:00:22.127948 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.127931    4550 scope.go:122] "RemoveContainer" containerID="f463314bbcebf34698adab87240311464e442dcda902c8c7d24ccf68573ecec0"
Nov 03 01:00:22.128481 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:22.128411    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc_odh-ai-gateway-infra(deec6453-23fd-49d0-aded-4518c0e1b5e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2"
Nov 03 01:00:22.131029 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.131006    4550 scope.go:122] "RemoveContainer" containerID="d17f7078e904501b3f1a96bc6e765c0e994d05cb7b1cace9fca14ea4c065ee74"
Nov 03 01:00:22.135063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.134213    4550 generic.go:356] "Generic (PLEG): container finished" podID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerID="0df98fe8ad60cdf6f694e3bcf6f42b3fc9dabeaeb763a9fbfe78b7fb8462fdb3" exitCode=60
Nov 03 01:00:22.135063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.134293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" event={"ID":"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c","Type":"ContainerDied","Data":"0df98fe8ad60cdf6f694e3bcf6f42b3fc9dabeaeb763a9fbfe78b7fb8462fdb3"}
Nov 03 01:00:22.135063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.134691    4550 scope.go:122] "RemoveContainer" containerID="0df98fe8ad60cdf6f694e3bcf6f42b3fc9dabeaeb763a9fbfe78b7fb8462fdb3"
Nov 03 01:00:22.135063 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:22.135000    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk_odh-ai-gateway-infra(95e9e7b8-65ef-47db-bb3e-8a62be6eef4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c"
Nov 03 01:00:22.139485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.138863    4550 generic.go:356] "Generic (PLEG): container finished" podID="d9598928-b49d-40ab-9e70-e127b122c329" containerID="07e62286cb7f2c147a81af4de38321e0d4fe48dcd9bb301e6859514c0ad1bb0d" exitCode=60
Nov 03 01:00:22.139485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.138920    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerDied","Data":"07e62286cb7f2c147a81af4de38321e0d4fe48dcd9bb301e6859514c0ad1bb0d"}
Nov 03 01:00:22.139485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.139189    4550 scope.go:122] "RemoveContainer" containerID="07e62286cb7f2c147a81af4de38321e0d4fe48dcd9bb301e6859514c0ad1bb0d"
Nov 03 01:00:22.139485 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:22.139410    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl_odh-ai-gateway-infra(d9598928-b49d-40ab-9e70-e127b122c329)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" podUID="d9598928-b49d-40ab-9e70-e127b122c329"
Nov 03 01:00:22.145913 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.145848    4550 generic.go:356] "Generic (PLEG): container finished" podID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerID="4562587d91907605d67c8ce7ee4ae02defafbc73161d5f3c9221faa41fe64421" exitCode=60
Nov 03 01:00:22.146167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.146124    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerDied","Data":"4562587d91907605d67c8ce7ee4ae02defafbc73161d5f3c9221faa41fe64421"}
Nov 03 01:00:22.146594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.146567    4550 scope.go:122] "RemoveContainer" containerID="4562587d91907605d67c8ce7ee4ae02defafbc73161d5f3c9221faa41fe64421"
Nov 03 01:00:22.146959 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:22.146871    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh_odh-ai-gateway-infra(34a5ddfa-213d-472a-9460-3e770c9cb1a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8"
Nov 03 01:00:22.153070 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.153035    4550 scope.go:122] "RemoveContainer" containerID="d5ad885014533b398b15d27aa26e4146d21f11a954ba0f159cd1f9fa0b0c954c"
Nov 03 01:00:22.155543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.154688    4550 generic.go:356] "Generic (PLEG): container finished" podID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerID="e534aa4299646cb533240cac379b599207aa61fb6b0aee9a5c49b7c9d03ef655" exitCode=60
Nov 03 01:00:22.155543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.154781    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerDied","Data":"e534aa4299646cb533240cac379b599207aa61fb6b0aee9a5c49b7c9d03ef655"}
Nov 03 01:00:22.155543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.155139    4550 scope.go:122] "RemoveContainer" containerID="e534aa4299646cb533240cac379b599207aa61fb6b0aee9a5c49b7c9d03ef655"
Nov 03 01:00:22.155543 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:22.155484    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp_odh-ai-gateway-infra(e70efcea-af98-4d4a-bd08-490cb86f055b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b"
Nov 03 01:00:22.167889 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.163716    4550 generic.go:356] "Generic (PLEG): container finished" podID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerID="0a6f1e1f6d2003c7ec4dd694c047b7fb5091c12beb33e43a537d344eadd36c35" exitCode=60
Nov 03 01:00:22.167889 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.163819    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" event={"ID":"aa394189-738e-4dc7-8cef-c0576e53a98d","Type":"ContainerDied","Data":"0a6f1e1f6d2003c7ec4dd694c047b7fb5091c12beb33e43a537d344eadd36c35"}
Nov 03 01:00:22.172660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.172631    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerStarted","Data":"740b4f1cce807d8bc7f36af9c421b3ae252491fc22e143a4a2eca94402ac8026"}
Nov 03 01:00:22.183107 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.183011    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerStarted","Data":"d0afe9dafb03dafa9adcdba0667db7436181cef1555b3c0b9ef0ee284376df55"}
Nov 03 01:00:22.184767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.184716    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx" event={"ID":"90507494-4818-4eb4-a1cb-a3e682b3bdd8","Type":"ContainerDied","Data":"7808f577dd239e100183785b9b37a9e5feb60725378c06547758a5e1be2e2bfe"}
Nov 03 01:00:22.184865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.184808    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"
Nov 03 01:00:22.259716 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.240117    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"]
Nov 03 01:00:22.259716 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.252720    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-883a9e-29894460-ckng6"]
Nov 03 01:00:22.373460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.373351    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" podStartSLOduration=22.373334265 podStartE2EDuration="22.373334265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:22.364994715 +0000 UTC m=+3304.686531518" watchObservedRunningTime="2026-11-03 01:00:22.373334265 +0000 UTC m=+3304.694871062"
Nov 03 01:00:22.421638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.421502    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"]
Nov 03 01:00:22.439204 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.439172    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"]
Nov 03 01:00:22.441316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.441006    4550 scope.go:122] "RemoveContainer" containerID="c19d4a9fee7405c1a5249f75a192ed58598858438402938a87b7f87b8b5ff634"
Nov 03 01:00:22.457886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.457848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:22.471393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.471368    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"]
Nov 03 01:00:22.598731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.598633    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:22.665675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.665641    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz\" (UniqueName: \"kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz\") pod \"f53c2794-70d3-497e-91f6-08e4ec16bf59\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") "
Nov 03 01:00:22.672371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.672339    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca\") pod \"f53c2794-70d3-497e-91f6-08e4ec16bf59\" (UID: \"f53c2794-70d3-497e-91f6-08e4ec16bf59\") "
Nov 03 01:00:22.672939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.672404    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz" pod "f53c2794-70d3-497e-91f6-08e4ec16bf59" (UID: "f53c2794-70d3-497e-91f6-08e4ec16bf59"). InnerVolumeSpecName "kube-api-access-2k6vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:22.672939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.672762    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca" pod "f53c2794-70d3-497e-91f6-08e4ec16bf59" (UID: "f53c2794-70d3-497e-91f6-08e4ec16bf59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:22.678031 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.678005    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f53c2794-70d3-497e-91f6-08e4ec16bf59-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:22.678180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.678166    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6vz\" (UniqueName: \"kubernetes.io/projected/f53c2794-70d3-497e-91f6-08e4ec16bf59-kube-api-access-2k6vz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:22.743129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.743019    4550 scope.go:122] "RemoveContainer" containerID="04fb01e23017d6e60104701f45cdddbbe1af5def81eafd7217dd9c9b471e75e4"
Nov 03 01:00:22.762410 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.762378    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"]
Nov 03 01:00:22.773261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.773177    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"]
Nov 03 01:00:22.780693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.780660    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:22.803634 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.803577    4550 scope.go:122] "RemoveContainer" containerID="70ba3947fd3f8f0fca653dd4d3e86b8c440f3b24127f967de094d4a5ff38892e"
Nov 03 01:00:22.825111 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.824603    4550 scope.go:122] "RemoveContainer" containerID="d18f99633c0015491dc052efce637fda4440f4ca5743c66b76b3c82cb5bf4130"
Nov 03 01:00:22.856544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.855403    4550 scope.go:122] "RemoveContainer" containerID="9a550c63e260e92043646e34422df089ffcc9c48ba7765a20e02b8ea91f6e35f"
Nov 03 01:00:22.891736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.891709    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw\" (UniqueName: \"kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw\") pod \"4171a7bd-72e5-4863-9cdb-5637237dcddf\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") "
Nov 03 01:00:22.891938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.891920    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca\") pod \"4171a7bd-72e5-4863-9cdb-5637237dcddf\" (UID: \"4171a7bd-72e5-4863-9cdb-5637237dcddf\") "
Nov 03 01:00:22.893757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.893739    4550 scope.go:122] "RemoveContainer" containerID="a24fddee34f2055e9e65c0789e352991c039a18f5bc2414e9ec9b440dbd179b4"
Nov 03 01:00:22.895910 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.895824    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca" pod "4171a7bd-72e5-4863-9cdb-5637237dcddf" (UID: "4171a7bd-72e5-4863-9cdb-5637237dcddf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:22.899973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.899942    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw" pod "4171a7bd-72e5-4863-9cdb-5637237dcddf" (UID: "4171a7bd-72e5-4863-9cdb-5637237dcddf"). InnerVolumeSpecName "kube-api-access-bz6vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:22.932975 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.932703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:22.934862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.934687    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"]
Nov 03 01:00:22.944108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.944011    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d4d8c2-29894460-qqg8c"]
Nov 03 01:00:22.973346 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.973265    4550 scope.go:122] "RemoveContainer" containerID="edabe2f374efe1b1762378d4b6a71c70bcdcd3885707be5449b230222c0d0edd"
Nov 03 01:00:22.979066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.978991    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:22.981127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.981102    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:22.987241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.987189    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:22.992607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.992592    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.994036    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw\" (UniqueName: \"kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw\") pod \"068293c7-3cf2-4d37-94c7-958ca42a55f5\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") "
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.994110    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx\" (UniqueName: \"kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx\") pod \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") "
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.994146    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca\") pod \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") "
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.994249    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca\") pod \"068293c7-3cf2-4d37-94c7-958ca42a55f5\" (UID: \"068293c7-3cf2-4d37-94c7-958ca42a55f5\") "
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.994316    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq\" (UniqueName: \"kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq\") pod \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\" (UID: \"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab\") "
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.994391    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca\") pod \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\" (UID: \"36a4e5e2-d23b-4006-a3fa-448ded9b054f\") "
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.995150    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz6vw\" (UniqueName: \"kubernetes.io/projected/4171a7bd-72e5-4863-9cdb-5637237dcddf-kube-api-access-bz6vw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:22.995411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.995173    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4171a7bd-72e5-4863-9cdb-5637237dcddf-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:22.996285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.996028    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca" pod "36a4e5e2-d23b-4006-a3fa-448ded9b054f" (UID: "36a4e5e2-d23b-4006-a3fa-448ded9b054f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:22.996858 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.996836    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca" pod "068293c7-3cf2-4d37-94c7-958ca42a55f5" (UID: "068293c7-3cf2-4d37-94c7-958ca42a55f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:22.998005 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.997977    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca" pod "620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" (UID: "620eee4a-a44a-45ce-8cdc-b9a4368aa6ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:22.999890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:22.999868    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx" pod "36a4e5e2-d23b-4006-a3fa-448ded9b054f" (UID: "36a4e5e2-d23b-4006-a3fa-448ded9b054f"). InnerVolumeSpecName "kube-api-access-scppx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.003354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.003336    4550 scope.go:122] "RemoveContainer" containerID="d03ee6c549e370f78360e164c4578db17a4024df1ce9052420df353dd4dee823"
Nov 03 01:00:23.003659 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.003637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw" pod "068293c7-3cf2-4d37-94c7-958ca42a55f5" (UID: "068293c7-3cf2-4d37-94c7-958ca42a55f5"). InnerVolumeSpecName "kube-api-access-mlrqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.004602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.004571    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq" pod "620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" (UID: "620eee4a-a44a-45ce-8cdc-b9a4368aa6ab"). InnerVolumeSpecName "kube-api-access-rqdxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.049474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.049452    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:23.095747 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.095707    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp\" (UniqueName: \"kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp\") pod \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") "
Nov 03 01:00:23.095891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.095761    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v\" (UniqueName: \"kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v\") pod \"aa394189-738e-4dc7-8cef-c0576e53a98d\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") "
Nov 03 01:00:23.095983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.095966    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca\") pod \"aa394189-738e-4dc7-8cef-c0576e53a98d\" (UID: \"aa394189-738e-4dc7-8cef-c0576e53a98d\") "
Nov 03 01:00:23.096058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096002    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca\") pod \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\" (UID: \"9e0a52de-9acc-4c3e-bb27-fa6137fee5df\") "
Nov 03 01:00:23.096461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096399    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca" pod "aa394189-738e-4dc7-8cef-c0576e53a98d" (UID: "aa394189-738e-4dc7-8cef-c0576e53a98d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.096590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096567    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca" pod "9e0a52de-9acc-4c3e-bb27-fa6137fee5df" (UID: "9e0a52de-9acc-4c3e-bb27-fa6137fee5df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.096744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096722    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa394189-738e-4dc7-8cef-c0576e53a98d-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096739    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096749    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36a4e5e2-d23b-4006-a3fa-448ded9b054f-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096758    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrqw\" (UniqueName: \"kubernetes.io/projected/068293c7-3cf2-4d37-94c7-958ca42a55f5-kube-api-access-mlrqw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096768    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scppx\" (UniqueName: \"kubernetes.io/projected/36a4e5e2-d23b-4006-a3fa-448ded9b054f-kube-api-access-scppx\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096777    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096788    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/068293c7-3cf2-4d37-94c7-958ca42a55f5-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.096980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.096802    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqdxq\" (UniqueName: \"kubernetes.io/projected/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab-kube-api-access-rqdxq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.098104 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.098081    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp" pod "9e0a52de-9acc-4c3e-bb27-fa6137fee5df" (UID: "9e0a52de-9acc-4c3e-bb27-fa6137fee5df"). InnerVolumeSpecName "kube-api-access-2qbvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.098525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.098508    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v" pod "aa394189-738e-4dc7-8cef-c0576e53a98d" (UID: "aa394189-738e-4dc7-8cef-c0576e53a98d"). InnerVolumeSpecName "kube-api-access-v528v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.138725 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.138700    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"]
Nov 03 01:00:23.144718 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.144689    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894460-rhfxx"]
Nov 03 01:00:23.198292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.198265    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qbvp\" (UniqueName: \"kubernetes.io/projected/9e0a52de-9acc-4c3e-bb27-fa6137fee5df-kube-api-access-2qbvp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.198292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.198291    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v528v\" (UniqueName: \"kubernetes.io/projected/aa394189-738e-4dc7-8cef-c0576e53a98d-kube-api-access-v528v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.200832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.200808    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"
Nov 03 01:00:23.200947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.200837    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj" event={"ID":"aa394189-738e-4dc7-8cef-c0576e53a98d","Type":"ContainerDied","Data":"d4a94d8999cceeecee1298504ac66004e614ec57c6baaa14aa677419f2a9a6a9"}
Nov 03 01:00:23.200947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.200923    4550 scope.go:122] "RemoveContainer" containerID="0a6f1e1f6d2003c7ec4dd694c047b7fb5091c12beb33e43a537d344eadd36c35"
Nov 03 01:00:23.203210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.203182    4550 generic.go:356] "Generic (PLEG): container finished" podID="4ca04320-6331-4640-b213-c761707a2024" containerID="740b4f1cce807d8bc7f36af9c421b3ae252491fc22e143a4a2eca94402ac8026" exitCode=60
Nov 03 01:00:23.203325 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.203254    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerDied","Data":"740b4f1cce807d8bc7f36af9c421b3ae252491fc22e143a4a2eca94402ac8026"}
Nov 03 01:00:23.203709 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.203690    4550 scope.go:122] "RemoveContainer" containerID="740b4f1cce807d8bc7f36af9c421b3ae252491fc22e143a4a2eca94402ac8026"
Nov 03 01:00:23.204026 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:23.204003    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d_odh-ai-gateway-infra(4ca04320-6331-4640-b213-c761707a2024)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" podUID="4ca04320-6331-4640-b213-c761707a2024"
Nov 03 01:00:23.205803 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.205783    4550 generic.go:356] "Generic (PLEG): container finished" podID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerID="d0afe9dafb03dafa9adcdba0667db7436181cef1555b3c0b9ef0ee284376df55" exitCode=60
Nov 03 01:00:23.205928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.205859    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerDied","Data":"d0afe9dafb03dafa9adcdba0667db7436181cef1555b3c0b9ef0ee284376df55"}
Nov 03 01:00:23.206218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.206200    4550 scope.go:122] "RemoveContainer" containerID="d0afe9dafb03dafa9adcdba0667db7436181cef1555b3c0b9ef0ee284376df55"
Nov 03 01:00:23.206540 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:23.206521    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-7bhb4_odh-ai-gateway-infra(c3442fb4-b070-4d02-b0a7-0f78e83f5c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61"
Nov 03 01:00:23.208681 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.208659    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"
Nov 03 01:00:23.208786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.208665    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh" event={"ID":"36a4e5e2-d23b-4006-a3fa-448ded9b054f","Type":"ContainerDied","Data":"87ccddcb4e3a240016db87b9dc5796bb1b23ab03ae38312e4d0234be7cef87bf"}
Nov 03 01:00:23.210219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.210189    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"
Nov 03 01:00:23.210695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.210569    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln" event={"ID":"f53c2794-70d3-497e-91f6-08e4ec16bf59","Type":"ContainerDied","Data":"0745e07993000737168c5bf6637945b015451dc7994d6b0e18429b13492f20fa"}
Nov 03 01:00:23.232387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.232363    4550 scope.go:122] "RemoveContainer" containerID="d69e30e9ca87ea4940e5e150fc9f080a2f57564bc0de49df7122b778f65d686e"
Nov 03 01:00:23.234635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.234227    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"
Nov 03 01:00:23.234635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.234167    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx" event={"ID":"4171a7bd-72e5-4863-9cdb-5637237dcddf","Type":"ContainerDied","Data":"7c1bc63f8380e129b24700593d87a2c926bb8c88274649835c0d697566d9cba6"}
Nov 03 01:00:23.239331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.239315    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"
Nov 03 01:00:23.240139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.239318    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp" event={"ID":"620eee4a-a44a-45ce-8cdc-b9a4368aa6ab","Type":"ContainerDied","Data":"fb81cc62f409d9cda6643ef403d5b2638c3f82e28f84e904e56dd6438ddae22f"}
Nov 03 01:00:23.242119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.242096    4550 generic.go:356] "Generic (PLEG): container finished" podID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerID="26baf6a9f42af3687c062a03d2f2dea19d0e58aebc9d56974a55510cab62e142" exitCode=60
Nov 03 01:00:23.242227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.242209    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerDied","Data":"26baf6a9f42af3687c062a03d2f2dea19d0e58aebc9d56974a55510cab62e142"}
Nov 03 01:00:23.242683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.242666    4550 scope.go:122] "RemoveContainer" containerID="26baf6a9f42af3687c062a03d2f2dea19d0e58aebc9d56974a55510cab62e142"
Nov 03 01:00:23.243070 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:23.243019    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j_odh-ai-gateway-infra(540463f3-b44b-4da2-adeb-192f817fcf3c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c"
Nov 03 01:00:23.244723 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.244700    4550 generic.go:356] "Generic (PLEG): container finished" podID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerID="143de626a421cb0e5d7d476bfdacdc4af5b741859f946df16c14456c11272d6d" exitCode=60
Nov 03 01:00:23.244813 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.244789    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" event={"ID":"8f332463-51f7-4d57-ae51-99f46e0074b6","Type":"ContainerDied","Data":"143de626a421cb0e5d7d476bfdacdc4af5b741859f946df16c14456c11272d6d"}
Nov 03 01:00:23.245247 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.245206    4550 scope.go:122] "RemoveContainer" containerID="143de626a421cb0e5d7d476bfdacdc4af5b741859f946df16c14456c11272d6d"
Nov 03 01:00:23.245585 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:23.245551    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n_odh-ai-gateway-infra(8f332463-51f7-4d57-ae51-99f46e0074b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6"
Nov 03 01:00:23.257256 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.257230    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn" event={"ID":"9e0a52de-9acc-4c3e-bb27-fa6137fee5df","Type":"ContainerDied","Data":"d84d34e6ca6df5223ae2afaaf90e5af02f292fe5a412ac0e756eb8dc262b7a70"}
Nov 03 01:00:23.257378 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.257314    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"
Nov 03 01:00:23.259612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.259595    4550 scope.go:122] "RemoveContainer" containerID="2d913c407c4417bcb19ba4c21ec89aa6cdfffd889cb8e94d069e625e6a9d6dd5"
Nov 03 01:00:23.276911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.276889    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc" event={"ID":"068293c7-3cf2-4d37-94c7-958ca42a55f5","Type":"ContainerDied","Data":"e1b75de87a3241f525c00309c9bcd113571346950036eeadc892bac692e6f6fb"}
Nov 03 01:00:23.280302 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.277097    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"
Nov 03 01:00:23.281138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.280173    4550 scope.go:122] "RemoveContainer" containerID="b1e78289e9d4a65c2e73f09901acce79c49ebfb8acc62bb846aa9c28865ecd39"
Nov 03 01:00:23.323994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.322270    4550 scope.go:122] "RemoveContainer" containerID="0c0f0cb3b90f356ce6573fa3cc5f56a654ab39e9cebba190f3896b6ddeaa3672"
Nov 03 01:00:23.335085 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.332105    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"]
Nov 03 01:00:23.382156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.382123    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"]
Nov 03 01:00:23.382619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.382595    4550 scope.go:122] "RemoveContainer" containerID="3de797920c4cf25d68828a35aaa5e0ac17edecadf9663f862fbb66cbd236ef68"
Nov 03 01:00:23.418013 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.417559    4550 scope.go:122] "RemoveContainer" containerID="1bd586f63182737faf68a49920c71bbef63d81cfc38c83c8f4fe9898fb8518fd"
Nov 03 01:00:23.427165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.427139    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:23.431593 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.431574    4550 scope.go:122] "RemoveContainer" containerID="8b43cf72fd76bbcb61327916776ed35743dcc8a0770824d580cd83ec981758ad"
Nov 03 01:00:23.458240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.458218    4550 scope.go:122] "RemoveContainer" containerID="0b3e2f751cf36128879a8a713b459fbe1637a95d7834682268a2347cad8a3c43"
Nov 03 01:00:23.478050 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.477987    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:23.508285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.490390    4550 scope.go:122] "RemoveContainer" containerID="c4811f98eb3f8aa79dae8d2413f70dc62d146aea7a1eca5a23f3191384fa40fb"
Nov 03 01:00:23.508285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.490522    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"]
Nov 03 01:00:23.539061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.530377    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n\" (UniqueName: \"kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n\") pod \"e70efcea-af98-4d4a-bd08-490cb86f055b\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") "
Nov 03 01:00:23.539764 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.539706    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca\") pod \"e70efcea-af98-4d4a-bd08-490cb86f055b\" (UID: \"e70efcea-af98-4d4a-bd08-490cb86f055b\") "
Nov 03 01:00:23.541222 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.541197    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca" pod "e70efcea-af98-4d4a-bd08-490cb86f055b" (UID: "e70efcea-af98-4d4a-bd08-490cb86f055b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.553594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.553537    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n" pod "e70efcea-af98-4d4a-bd08-490cb86f055b" (UID: "e70efcea-af98-4d4a-bd08-490cb86f055b"). InnerVolumeSpecName "kube-api-access-lrz5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.554025 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.553973    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-056a96-29894460-nw6mp"]
Nov 03 01:00:23.563055 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.562318    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"]
Nov 03 01:00:23.598402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.594052    4550 scope.go:122] "RemoveContainer" containerID="ecd8f0f04d31357a35088d142ff3888920bdd31ddb094392a68651524faf37b9"
Nov 03 01:00:23.629443 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.629398    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"]
Nov 03 01:00:23.639490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.637492    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:23.642151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.642126    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70efcea-af98-4d4a-bd08-490cb86f055b-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.642308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.642294    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrz5n\" (UniqueName: \"kubernetes.io/projected/e70efcea-af98-4d4a-bd08-490cb86f055b-kube-api-access-lrz5n\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.643605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.643579    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894460-svwln"]
Nov 03 01:00:23.709350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.709147    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"]
Nov 03 01:00:23.715151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.715124    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894460-zfpxh"]
Nov 03 01:00:23.718291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.718259    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:23.723189 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.723161    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:23.728603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.728586    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:23.744660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.743126    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh\" (UniqueName: \"kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh\") pod \"deec6453-23fd-49d0-aded-4518c0e1b5e2\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") "
Nov 03 01:00:23.744660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.743355    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca\") pod \"deec6453-23fd-49d0-aded-4518c0e1b5e2\" (UID: \"deec6453-23fd-49d0-aded-4518c0e1b5e2\") "
Nov 03 01:00:23.744660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.744051    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca" pod "deec6453-23fd-49d0-aded-4518c0e1b5e2" (UID: "deec6453-23fd-49d0-aded-4518c0e1b5e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.744660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.744240    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/deec6453-23fd-49d0-aded-4518c0e1b5e2-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.748268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.748244    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh" pod "deec6453-23fd-49d0-aded-4518c0e1b5e2" (UID: "deec6453-23fd-49d0-aded-4518c0e1b5e2"). InnerVolumeSpecName "kube-api-access-qk7rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.794883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.794807    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"]
Nov 03 01:00:23.839953 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.839923    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" path="/var/lib/kubelet/pods/1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce/volumes"
Nov 03 01:00:23.840545 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.840525    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" path="/var/lib/kubelet/pods/30c9afba-79f5-41d3-ae1e-0cf85863e6d6/volumes"
Nov 03 01:00:23.841178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.841154    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" path="/var/lib/kubelet/pods/36a4e5e2-d23b-4006-a3fa-448ded9b054f/volumes"
Nov 03 01:00:23.842030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.842011    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" path="/var/lib/kubelet/pods/620eee4a-a44a-45ce-8cdc-b9a4368aa6ab/volumes"
Nov 03 01:00:23.842555 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.842538    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" path="/var/lib/kubelet/pods/90507494-4818-4eb4-a1cb-a3e682b3bdd8/volumes"
Nov 03 01:00:23.843052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.843026    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" path="/var/lib/kubelet/pods/f53c2794-70d3-497e-91f6-08e4ec16bf59/volumes"
Nov 03 01:00:23.845362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845297    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:23.845476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845380    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn\" (UniqueName: \"kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn\") pod \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") "
Nov 03 01:00:23.845476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845423    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7\" (UniqueName: \"kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7\") pod \"d9598928-b49d-40ab-9e70-e127b122c329\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") "
Nov 03 01:00:23.845638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845621    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca\") pod \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\" (UID: \"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c\") "
Nov 03 01:00:23.845683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845667    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca\") pod \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") "
Nov 03 01:00:23.845751 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845736    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca\") pod \"d9598928-b49d-40ab-9e70-e127b122c329\" (UID: \"d9598928-b49d-40ab-9e70-e127b122c329\") "
Nov 03 01:00:23.845842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.845823    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg\" (UniqueName: \"kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg\") pod \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\" (UID: \"34a5ddfa-213d-472a-9460-3e770c9cb1a8\") "
Nov 03 01:00:23.846064 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.846044    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca" pod "95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" (UID: "95e9e7b8-65ef-47db-bb3e-8a62be6eef4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.846502 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.846476    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.846618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.846506    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk7rh\" (UniqueName: \"kubernetes.io/projected/deec6453-23fd-49d0-aded-4518c0e1b5e2-kube-api-access-qk7rh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.847377 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.847352    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca" pod "34a5ddfa-213d-472a-9460-3e770c9cb1a8" (UID: "34a5ddfa-213d-472a-9460-3e770c9cb1a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.847650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.847630    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca" pod "d9598928-b49d-40ab-9e70-e127b122c329" (UID: "d9598928-b49d-40ab-9e70-e127b122c329"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.849031 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.849006    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn" pod "95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" (UID: "95e9e7b8-65ef-47db-bb3e-8a62be6eef4c"). InnerVolumeSpecName "kube-api-access-fn4bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.849156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.849112    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7" pod "d9598928-b49d-40ab-9e70-e127b122c329" (UID: "d9598928-b49d-40ab-9e70-e127b122c329"). InnerVolumeSpecName "kube-api-access-kvws7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.850053 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.850036    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"]
Nov 03 01:00:23.850135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.850066    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-4c61c8-29894460-wh8fn"]
Nov 03 01:00:23.850525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.850501    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"]
Nov 03 01:00:23.850914 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.850892    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg" pod "34a5ddfa-213d-472a-9460-3e770c9cb1a8" (UID: "34a5ddfa-213d-472a-9460-3e770c9cb1a8"). InnerVolumeSpecName "kube-api-access-2bxxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.855173 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.855152    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894460-v8bpx"]
Nov 03 01:00:23.947583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.947556    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx\" (UniqueName: \"kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx\") pod \"03449032-8806-4415-a9ff-7ac5f37df4ab\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") "
Nov 03 01:00:23.947777 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.947760    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca\") pod \"03449032-8806-4415-a9ff-7ac5f37df4ab\" (UID: \"03449032-8806-4415-a9ff-7ac5f37df4ab\") "
Nov 03 01:00:23.948158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948135    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca" pod "03449032-8806-4415-a9ff-7ac5f37df4ab" (UID: "03449032-8806-4415-a9ff-7ac5f37df4ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.948529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948511    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvws7\" (UniqueName: \"kubernetes.io/projected/d9598928-b49d-40ab-9e70-e127b122c329-kube-api-access-kvws7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.948582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948532    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03449032-8806-4415-a9ff-7ac5f37df4ab-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.948582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948541    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34a5ddfa-213d-472a-9460-3e770c9cb1a8-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.948582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948550    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9598928-b49d-40ab-9e70-e127b122c329-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.948582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948559    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxxg\" (UniqueName: \"kubernetes.io/projected/34a5ddfa-213d-472a-9460-3e770c9cb1a8-kube-api-access-2bxxg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.948582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.948567    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn4bn\" (UniqueName: \"kubernetes.io/projected/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c-kube-api-access-fn4bn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.949733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:23.949710    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx" pod "03449032-8806-4415-a9ff-7ac5f37df4ab" (UID: "03449032-8806-4415-a9ff-7ac5f37df4ab"). InnerVolumeSpecName "kube-api-access-bxjzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:24.027484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.027452    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"]
Nov 03 01:00:24.041292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.041262    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"]
Nov 03 01:00:24.050571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.050545    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxjzx\" (UniqueName: \"kubernetes.io/projected/03449032-8806-4415-a9ff-7ac5f37df4ab-kube-api-access-bxjzx\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.053711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.053683    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-359f71-29894460-vlgzj"]
Nov 03 01:00:24.233683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.233423    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"]
Nov 03 01:00:24.240702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.240673    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894460-rnbrc"]
Nov 03 01:00:24.313476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.313407    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"]
Nov 03 01:00:24.317583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.317515    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc" event={"ID":"03449032-8806-4415-a9ff-7ac5f37df4ab","Type":"ContainerDied","Data":"6dac84ea238a9703105ef13f6a5b33b7b48b371ee5b33192cf4e6fe151325e26"}
Nov 03 01:00:24.317815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.317748    4550 scope.go:122] "RemoveContainer" containerID="1ad86fed92b0bbfba8afa5b162c152ee8e5d6f1a9060badd473401dbdbf98c5f"
Nov 03 01:00:24.318702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.318422    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"
Nov 03 01:00:24.360762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.360699    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc" event={"ID":"deec6453-23fd-49d0-aded-4518c0e1b5e2","Type":"ContainerDied","Data":"b8a0a9f1b5cac9ab171d7ba6199f59d3fc4f0b85f74a21909e88acf6f0deecf8"}
Nov 03 01:00:24.361062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.361026    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"
Nov 03 01:00:24.368883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.368807    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk" event={"ID":"95e9e7b8-65ef-47db-bb3e-8a62be6eef4c","Type":"ContainerDied","Data":"21878433db2f18a300427880122ff172f692123632674ced8bfd25c492829cf1"}
Nov 03 01:00:24.369368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.369343    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"
Nov 03 01:00:24.378529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.372987    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl" event={"ID":"d9598928-b49d-40ab-9e70-e127b122c329","Type":"ContainerDied","Data":"b8643cd16e18192114c158f10356387364f3368250a884a063b2b92e03bfd586"}
Nov 03 01:00:24.378529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.373094    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"
Nov 03 01:00:24.382992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.382968    4550 scope.go:122] "RemoveContainer" containerID="f463314bbcebf34698adab87240311464e442dcda902c8c7d24ccf68573ecec0"
Nov 03 01:00:24.410413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.410391    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"
Nov 03 01:00:24.418712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.418692    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"
Nov 03 01:00:24.419585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.410642    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh" event={"ID":"34a5ddfa-213d-472a-9460-3e770c9cb1a8","Type":"ContainerDied","Data":"fc7d5d91f83bde547209ea8cbca9969300f77d4c4ba705be201aa8dcf968801b"}
Nov 03 01:00:24.419808 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.419775    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp" event={"ID":"e70efcea-af98-4d4a-bd08-490cb86f055b","Type":"ContainerDied","Data":"30354669139d39011abda1857909ae48e1b215c7b48c38637a02862735b13abe"}
Nov 03 01:00:24.460369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.460347    4550 scope.go:122] "RemoveContainer" containerID="0df98fe8ad60cdf6f694e3bcf6f42b3fc9dabeaeb763a9fbfe78b7fb8462fdb3"
Nov 03 01:00:24.477130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.477107    4550 scope.go:122] "RemoveContainer" containerID="07e62286cb7f2c147a81af4de38321e0d4fe48dcd9bb301e6859514c0ad1bb0d"
Nov 03 01:00:24.492837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.492818    4550 scope.go:122] "RemoveContainer" containerID="4562587d91907605d67c8ce7ee4ae02defafbc73161d5f3c9221faa41fe64421"
Nov 03 01:00:24.514244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.514220    4550 scope.go:122] "RemoveContainer" containerID="e534aa4299646cb533240cac379b599207aa61fb6b0aee9a5c49b7c9d03ef655"
Nov 03 01:00:24.581984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.581925    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"]
Nov 03 01:00:24.588743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.588712    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894460-lmslp"]
Nov 03 01:00:24.626344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.626284    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"]
Nov 03 01:00:24.638158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.638131    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-726205-29894460-p2hrc"]
Nov 03 01:00:24.656022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.656002    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:24.659633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.659612    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:24.663972 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.663953    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:24.665983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.665458    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg\" (UniqueName: \"kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg\") pod \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") "
Nov 03 01:00:24.665983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.665568    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca\") pod \"4ca04320-6331-4640-b213-c761707a2024\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") "
Nov 03 01:00:24.665983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.665677    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca\") pod \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\" (UID: \"c3442fb4-b070-4d02-b0a7-0f78e83f5c61\") "
Nov 03 01:00:24.665983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.665783    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz\" (UniqueName: \"kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz\") pod \"4ca04320-6331-4640-b213-c761707a2024\" (UID: \"4ca04320-6331-4640-b213-c761707a2024\") "
Nov 03 01:00:24.665983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.665947    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca" pod "4ca04320-6331-4640-b213-c761707a2024" (UID: "4ca04320-6331-4640-b213-c761707a2024"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:24.666411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.666393    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca04320-6331-4640-b213-c761707a2024-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.669218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.667936    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca" pod "c3442fb4-b070-4d02-b0a7-0f78e83f5c61" (UID: "c3442fb4-b070-4d02-b0a7-0f78e83f5c61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:24.669733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.669708    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz" pod "4ca04320-6331-4640-b213-c761707a2024" (UID: "4ca04320-6331-4640-b213-c761707a2024"). InnerVolumeSpecName "kube-api-access-gtkzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:24.670104 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.670083    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:24.677169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.677149    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg" pod "c3442fb4-b070-4d02-b0a7-0f78e83f5c61" (UID: "c3442fb4-b070-4d02-b0a7-0f78e83f5c61"). InnerVolumeSpecName "kube-api-access-mkgmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:24.714525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.714496    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"]
Nov 03 01:00:24.717204 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.717184    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-84f605-29894460-d2jsl"]
Nov 03 01:00:24.742110 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:24.742073    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:00:24.749666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.749607    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"]
Nov 03 01:00:24.753721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.753702    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-49d7e5-29894460-fxplk"]
Nov 03 01:00:24.767191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.767169    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x\" (UniqueName: \"kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x\") pod \"8f332463-51f7-4d57-ae51-99f46e0074b6\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") "
Nov 03 01:00:24.767304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.767288    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca\") pod \"8f332463-51f7-4d57-ae51-99f46e0074b6\" (UID: \"8f332463-51f7-4d57-ae51-99f46e0074b6\") "
Nov 03 01:00:24.767381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.767366    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca\") pod \"540463f3-b44b-4da2-adeb-192f817fcf3c\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") "
Nov 03 01:00:24.767482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.767449    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p\") pod \"540463f3-b44b-4da2-adeb-192f817fcf3c\" (UID: \"540463f3-b44b-4da2-adeb-192f817fcf3c\") "
Nov 03 01:00:24.767722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.767692    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca" pod "8f332463-51f7-4d57-ae51-99f46e0074b6" (UID: "8f332463-51f7-4d57-ae51-99f46e0074b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:24.767722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.767714    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca" pod "540463f3-b44b-4da2-adeb-192f817fcf3c" (UID: "540463f3-b44b-4da2-adeb-192f817fcf3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:24.768138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.768113    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkgmg\" (UniqueName: \"kubernetes.io/projected/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-kube-api-access-mkgmg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.768138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.768141    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f332463-51f7-4d57-ae51-99f46e0074b6-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.768324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.768151    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/540463f3-b44b-4da2-adeb-192f817fcf3c-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.768324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.768160    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3442fb4-b070-4d02-b0a7-0f78e83f5c61-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.768324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.768169    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkzz\" (UniqueName: \"kubernetes.io/projected/4ca04320-6331-4640-b213-c761707a2024-kube-api-access-gtkzz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.769331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.769311    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x" pod "8f332463-51f7-4d57-ae51-99f46e0074b6" (UID: "8f332463-51f7-4d57-ae51-99f46e0074b6"). InnerVolumeSpecName "kube-api-access-zxl4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:24.769513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.769486    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p" pod "540463f3-b44b-4da2-adeb-192f817fcf3c" (UID: "540463f3-b44b-4da2-adeb-192f817fcf3c"). InnerVolumeSpecName "kube-api-access-24r9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:24.866831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.866804    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"]
Nov 03 01:00:24.869879 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.869853    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/540463f3-b44b-4da2-adeb-192f817fcf3c-kube-api-access-24r9p\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.870074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.870034    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxl4x\" (UniqueName: \"kubernetes.io/projected/8f332463-51f7-4d57-ae51-99f46e0074b6-kube-api-access-zxl4x\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:24.872191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.872173    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8ed55a-29894460-ghlvc"]
Nov 03 01:00:24.907271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.907222    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"]
Nov 03 01:00:24.913205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:24.913171    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894460-qsvgh"]
Nov 03 01:00:25.159529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.159495    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:25.433680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.433599    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"
Nov 03 01:00:25.433858 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.433595    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j" event={"ID":"540463f3-b44b-4da2-adeb-192f817fcf3c","Type":"ContainerDied","Data":"ff646c25fd724d3d8bf6425f308682fb0b0ee9a8f777ee7e91d44ab320753b6c"}
Nov 03 01:00:25.433858 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.433744    4550 scope.go:122] "RemoveContainer" containerID="26baf6a9f42af3687c062a03d2f2dea19d0e58aebc9d56974a55510cab62e142"
Nov 03 01:00:25.435342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.435324    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"
Nov 03 01:00:25.436030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.436005    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n" event={"ID":"8f332463-51f7-4d57-ae51-99f46e0074b6","Type":"ContainerDied","Data":"fe9cb76c787844a3e915b66c23f17763369172bb1ec74c37fd8f13a172c41d02"}
Nov 03 01:00:25.442857 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.442833    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d" event={"ID":"4ca04320-6331-4640-b213-c761707a2024","Type":"ContainerDied","Data":"16ded95182c7b679e92a641c1752d51ba77e87895565615891351dd67a18d65c"}
Nov 03 01:00:25.442971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.442847    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"
Nov 03 01:00:25.444393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.444360    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4" event={"ID":"c3442fb4-b070-4d02-b0a7-0f78e83f5c61","Type":"ContainerDied","Data":"f88b1f6e1b624f5c492625a1a8a564103aed8f5a800ff7507e1810cb4ad5c2a0"}
Nov 03 01:00:25.444524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.444371    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"
Nov 03 01:00:25.449745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.449725    4550 scope.go:122] "RemoveContainer" containerID="143de626a421cb0e5d7d476bfdacdc4af5b741859f946df16c14456c11272d6d"
Nov 03 01:00:25.464742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.464719    4550 scope.go:122] "RemoveContainer" containerID="740b4f1cce807d8bc7f36af9c421b3ae252491fc22e143a4a2eca94402ac8026"
Nov 03 01:00:25.474757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.474737    4550 scope.go:122] "RemoveContainer" containerID="d0afe9dafb03dafa9adcdba0667db7436181cef1555b3c0b9ef0ee284376df55"
Nov 03 01:00:25.504604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.504582    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"]
Nov 03 01:00:25.508639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.508620    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-29d09b-29894460-p7s4n"]
Nov 03 01:00:25.557598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.557577    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"]
Nov 03 01:00:25.563849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.563827    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894460-lv96j"]
Nov 03 01:00:25.594030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.594005    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"]
Nov 03 01:00:25.598281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.598254    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-fc9316-29894460-b7n5d"]
Nov 03 01:00:25.630341 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.630310    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"]
Nov 03 01:00:25.634249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.634226    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-7bhb4"]
Nov 03 01:00:25.810601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.810531    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" path="/var/lib/kubelet/pods/03449032-8806-4415-a9ff-7ac5f37df4ab/volumes"
Nov 03 01:00:25.811036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.811020    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" path="/var/lib/kubelet/pods/068293c7-3cf2-4d37-94c7-958ca42a55f5/volumes"
Nov 03 01:00:25.811494 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.811478    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" path="/var/lib/kubelet/pods/34a5ddfa-213d-472a-9460-3e770c9cb1a8/volumes"
Nov 03 01:00:25.812237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.812222    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" path="/var/lib/kubelet/pods/4171a7bd-72e5-4863-9cdb-5637237dcddf/volumes"
Nov 03 01:00:25.812757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.812742    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca04320-6331-4640-b213-c761707a2024" path="/var/lib/kubelet/pods/4ca04320-6331-4640-b213-c761707a2024/volumes"
Nov 03 01:00:25.813158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.813140    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" path="/var/lib/kubelet/pods/540463f3-b44b-4da2-adeb-192f817fcf3c/volumes"
Nov 03 01:00:25.813829 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.813813    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" path="/var/lib/kubelet/pods/8f332463-51f7-4d57-ae51-99f46e0074b6/volumes"
Nov 03 01:00:25.814212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.814198    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" path="/var/lib/kubelet/pods/95e9e7b8-65ef-47db-bb3e-8a62be6eef4c/volumes"
Nov 03 01:00:25.814612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.814600    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" path="/var/lib/kubelet/pods/9e0a52de-9acc-4c3e-bb27-fa6137fee5df/volumes"
Nov 03 01:00:25.815255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.815242    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" path="/var/lib/kubelet/pods/aa394189-738e-4dc7-8cef-c0576e53a98d/volumes"
Nov 03 01:00:25.815654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.815641    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" path="/var/lib/kubelet/pods/c3442fb4-b070-4d02-b0a7-0f78e83f5c61/volumes"
Nov 03 01:00:25.816017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.816006    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9598928-b49d-40ab-9e70-e127b122c329" path="/var/lib/kubelet/pods/d9598928-b49d-40ab-9e70-e127b122c329/volumes"
Nov 03 01:00:25.816678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.816665    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" path="/var/lib/kubelet/pods/deec6453-23fd-49d0-aded-4518c0e1b5e2/volumes"
Nov 03 01:00:25.817045 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:25.817033    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" path="/var/lib/kubelet/pods/e70efcea-af98-4d4a-bd08-490cb86f055b/volumes"
Nov 03 01:00:26.935589 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:26.935561    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:28.255156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:28.255125    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"]
Nov 03 01:00:28.388495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:28.386683    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:28.531511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:28.523713    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:28.727454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:28.723759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:28.996043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:28.996006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:30.629765 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:30.629726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:32.213229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:32.213196    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:33.255590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:33.255551    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerName="payload-pre-processing" containerID="cri-o://a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c" gracePeriod=25
Nov 03 01:00:33.913677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:33.913643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:34.628937 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:34.628893    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:00:34.873641 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:34.873597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:35.617150 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:35.617107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:37.298293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:37.298242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:38.964231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:38.964194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:40.578509 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:40.578469    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:42.356217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:42.356179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:43.920452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:43.916744    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:44.011486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:44.011451    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:44.113650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:44.113603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:44.628294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:44.628258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:00:45.750580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:45.750545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:47.539503 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:47.539470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:49.344014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:49.343977    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:49.444608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:49.444569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:51.139664 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:51.139625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:52.937083 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:52.937036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:53.563182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.563148    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 01:00:53.669148 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.669110    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume\") pod \"6fb62e20-9899-476b-a1a4-0db41756bbd1\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") "
Nov 03 01:00:53.669362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.669204    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz\" (UniqueName: \"kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz\") pod \"6fb62e20-9899-476b-a1a4-0db41756bbd1\" (UID: \"6fb62e20-9899-476b-a1a4-0db41756bbd1\") "
Nov 03 01:00:53.669706 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.669680    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume" pod "6fb62e20-9899-476b-a1a4-0db41756bbd1" (UID: "6fb62e20-9899-476b-a1a4-0db41756bbd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:53.670258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.670229    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fb62e20-9899-476b-a1a4-0db41756bbd1-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:53.671825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.671801    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz" pod "6fb62e20-9899-476b-a1a4-0db41756bbd1" (UID: "6fb62e20-9899-476b-a1a4-0db41756bbd1"). InnerVolumeSpecName "kube-api-access-tr7lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:53.771764 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.771680    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr7lz\" (UniqueName: \"kubernetes.io/projected/6fb62e20-9899-476b-a1a4-0db41756bbd1-kube-api-access-tr7lz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:00:53.867416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.867374    4550 generic.go:356] "Generic (PLEG): container finished" podID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerID="a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c" exitCode=0
Nov 03 01:00:53.867640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.867469    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"
Nov 03 01:00:53.867640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.867528    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" event={"ID":"6fb62e20-9899-476b-a1a4-0db41756bbd1","Type":"ContainerDied","Data":"a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c"}
Nov 03 01:00:53.867754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.867662    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9" event={"ID":"6fb62e20-9899-476b-a1a4-0db41756bbd1","Type":"ContainerDied","Data":"4475fe069bc4dd5118927b2bd2f0fc2c3cb5fc62f128669d5fb6ef6d216e683c"}
Nov 03 01:00:53.867754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.867692    4550 scope.go:122] "RemoveContainer" containerID="a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c"
Nov 03 01:00:53.880496 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.880470    4550 scope.go:122] "RemoveContainer" containerID="a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c"
Nov 03 01:00:53.880827 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:53.880805    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c\": container with ID starting with a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c not found: ID does not exist" containerID="a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c"
Nov 03 01:00:53.880924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.880837    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c"} err="failed to get container status \"a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c\": rpc error: code = NotFound desc = could not find container \"a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c\": container with ID starting with a4d375eac28b9d0c311def850bba3062733274d0392babd5e5392d66f0a0d85c not found: ID does not exist"
Nov 03 01:00:53.924492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.924410    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"]
Nov 03 01:00:53.937497 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:53.937463    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-84f605-54f997d46c-t9jt9"]
Nov 03 01:00:54.633341 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:54.633307    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:54.760417 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:00:54.760375    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:00:55.811866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:55.811834    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" path="/var/lib/kubelet/pods/6fb62e20-9899-476b-a1a4-0db41756bbd1/volumes"
Nov 03 01:00:56.335754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:56.335721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:58.126127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:58.126092    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:00:59.865340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:00:59.865288    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:01.486417 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:01.486381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:03.131540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:03.131505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:04.763265 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:04.763209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:06.535613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:06.535562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:08.158066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:08.158015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:09.768143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:09.768091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:11.166724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:11.166686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:12.118726 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:12.118687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:13.792861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:13.792822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:14.212023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:14.211983    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:15.297284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:15.297246    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:15.460359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:15.460327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:15.571639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:15.571545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:15.767552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:15.767511    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:16.369452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:16.369086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:16.826745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:16.826621    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:17.692456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:17.690091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:18.121052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:18.120944    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:19.828712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:19.828684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:20.894400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:20.894370    4550 scope.go:122] "RemoveContainer" containerID="189598308b8c8f0f66f0178830647e8280625ef95443d14715d313b84f7ed010"
Nov 03 01:01:20.907531 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:20.907507    4550 scope.go:122] "RemoveContainer" containerID="a8cec05a77d3802a11eca82804b18175852c4161fbffc0c702c0017c77452f8f"
Nov 03 01:01:21.532086 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:21.532053    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:22.826365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:22.826325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:22.975963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:22.975864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:23.340042 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:23.340009    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:23.842285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:23.842249    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:24.727515 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:01:24.727472    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:01:24.912145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:24.912043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:25.754633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:25.754598    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:27.475202 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:27.475171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:28.312680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:28.312648    4550 generic.go:356] "Generic (PLEG): container finished" podID="4d304809-13c7-4fff-8287-b781199a7e3d" containerID="713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850" exitCode=0
Nov 03 01:01:28.312883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:28.312719    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" event={"ID":"4d304809-13c7-4fff-8287-b781199a7e3d","Type":"ContainerDied","Data":"713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"}
Nov 03 01:01:28.313197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:28.313181    4550 scope.go:122] "RemoveContainer" containerID="713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"
Nov 03 01:01:28.315269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:28.315242    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:01:28.315359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:28.315311    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:01:28.315359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:28.315344    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:01:29.139677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.139643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:29.322738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.322699    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" event={"ID":"4d304809-13c7-4fff-8287-b781199a7e3d","Type":"ContainerStarted","Data":"46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77"}
Nov 03 01:01:29.505215 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.505131    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 01:01:29.506835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.506802    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 01:01:29.506974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.506860    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 01:01:29.807266 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.807179    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:29.918421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:29.918391    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:30.505179 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:30.505140    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 01:01:30.505667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:30.505208    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 01:01:30.551536 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:30.551490    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:30.911175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:30.911139    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"]
Nov 03 01:01:30.911632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:30.911601    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy" containerID="cri-o://38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc" gracePeriod=30
Nov 03 01:01:30.931076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:30.931045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:31.040552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:31.040517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:31.505462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:31.505397    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 01:01:31.505977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:31.505480    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 01:01:31.626549 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:31.626514    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 01:01:31.912667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:31.912634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:32.028400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:32.028366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:32.126751 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:32.126716    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:32.505006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:32.504960    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 01:01:32.505243 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:32.505038    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 01:01:32.852629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:32.852544    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:33.505166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:33.505117    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 01:01:33.505397 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:33.505183    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 01:01:34.505044 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:34.505003    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body=
Nov 03 01:01:34.505519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:34.505066    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused"
Nov 03 01:01:34.507836 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:34.507803    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:35.510781 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:35.510750    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 01:01:35.512026 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:35.512003    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 01:01:36.212146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.212116    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 01:01:36.261589 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261555    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.261798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261605    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.261798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261691    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.261798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261744    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.261798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261796    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.262074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261821    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.262074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.261871    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.262229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.262086    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.263251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.262518    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs\") pod \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\" (UID: \"e0b34cfb-64e8-4c33-8a72-bc447bab068d\") "
Nov 03 01:01:36.263251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263086    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.263510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263262    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.263578 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263516    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:36.263640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263622    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.263767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263706    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.263767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263731    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.263767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.263744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.273506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.273448    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:01:36.274084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.274038    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:36.274481 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.274444    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:36.278667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.275382    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt" pod "e0b34cfb-64e8-4c33-8a72-bc447bab068d" (UID: "e0b34cfb-64e8-4c33-8a72-bc447bab068d"). InnerVolumeSpecName "kube-api-access-kkrpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:36.345462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.345090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:36.385612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385568    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.385612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385602    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.385612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385613    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrpt\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-kube-api-access-kkrpt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.385612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385622    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.386002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385631    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.386002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385640    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.386002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.385649    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b34cfb-64e8-4c33-8a72-bc447bab068d-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:01:36.421329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.421292    4550 generic.go:356] "Generic (PLEG): container finished" podID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerID="38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc" exitCode=0
Nov 03 01:01:36.421577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.421343    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"
Nov 03 01:01:36.421577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.421357    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" event={"ID":"e0b34cfb-64e8-4c33-8a72-bc447bab068d","Type":"ContainerDied","Data":"38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc"}
Nov 03 01:01:36.421577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.421420    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq" event={"ID":"e0b34cfb-64e8-4c33-8a72-bc447bab068d","Type":"ContainerDied","Data":"fb5cb43c7b9e46144f0dc731da3e1b39dd5599ec5ad80c020063601f977a4ff5"}
Nov 03 01:01:36.421577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.421458    4550 scope.go:122] "RemoveContainer" containerID="38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc"
Nov 03 01:01:36.434221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.434195    4550 scope.go:122] "RemoveContainer" containerID="38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc"
Nov 03 01:01:36.434522 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:01:36.434502    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc\": container with ID starting with 38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc not found: ID does not exist" containerID="38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc"
Nov 03 01:01:36.434602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.434530    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc"} err="failed to get container status \"38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc\": rpc error: code = NotFound desc = could not find container \"38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc\": container with ID starting with 38adf398645e429f47e2ca97cd45096b14d949e7d3bdf2285642da6542e16dcc not found: ID does not exist"
Nov 03 01:01:36.480012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.479970    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"]
Nov 03 01:01:36.486886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:36.486832    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-726205-openshift-default-b6c76c5b5-hrxxq"]
Nov 03 01:01:37.811664 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:37.811632    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" path="/var/lib/kubelet/pods/e0b34cfb-64e8-4c33-8a72-bc447bab068d/volumes"
Nov 03 01:01:38.033925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:38.033881    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:39.644538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:39.644505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:41.398217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:41.398170    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:43.102575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:43.102534    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:44.728415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:44.728381    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:45.038648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:45.038557    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"]
Nov 03 01:01:45.051187 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:45.051151    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb45343f-smlgz"]
Nov 03 01:01:45.810084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:45.810045    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d78242-6d86-407a-b31c-ea2fcd3e61a6" path="/var/lib/kubelet/pods/23d78242-6d86-407a-b31c-ea2fcd3e61a6/volumes"
Nov 03 01:01:46.032642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:46.032597    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"]
Nov 03 01:01:46.045651 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:46.045616    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2be699cb-kw4cn"]
Nov 03 01:01:46.593343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:46.593294    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:47.811599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:47.811562    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab52aee-b962-4efa-8c44-6928bd93dbba" path="/var/lib/kubelet/pods/fab52aee-b962-4efa-8c44-6928bd93dbba/volumes"
Nov 03 01:01:48.420808 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:48.420759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:50.124205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:50.124172    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:50.818483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:50.817784    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:51.884565 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:51.877112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:52.944613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:52.944583    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:53.708751 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:53.708712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:54.728259 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:01:54.728230    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:01:55.525836 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:55.525802    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:56.038574 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.038536    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"]
Nov 03 01:01:56.047320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.047292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:56.049252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.049222    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"]
Nov 03 01:01:56.052877 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.052855    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f6dc7ab-76jjt"]
Nov 03 01:01:56.056120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.056095    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-007417d6-rf6ck"]
Nov 03 01:01:56.440714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.438617    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": read tcp 10.133.0.2:35322->10.133.0.241:15021: read: connection reset by peer" start-of-body=
Nov 03 01:01:56.440714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.438672    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": read tcp 10.133.0.2:35322->10.133.0.241:15021: read: connection reset by peer"
Nov 03 01:01:56.736782 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.736712    4550 generic.go:356] "Generic (PLEG): container finished" podID="faa828ae-d375-4578-9485-71ed468a41e1" containerID="ce701d4ad0fe7a60fd290e10f34560f442e2c3aaa49638b3940f937ea0af183c" exitCode=0
Nov 03 01:01:56.736931 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.736783    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" event={"ID":"faa828ae-d375-4578-9485-71ed468a41e1","Type":"ContainerDied","Data":"ce701d4ad0fe7a60fd290e10f34560f442e2c3aaa49638b3940f937ea0af183c"}
Nov 03 01:01:56.737172 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.737150    4550 scope.go:122] "RemoveContainer" containerID="ce701d4ad0fe7a60fd290e10f34560f442e2c3aaa49638b3940f937ea0af183c"
Nov 03 01:01:56.739204 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.739141    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:01:56.739294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.739260    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:01:56.739340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:56.739295    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:01:57.414711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:57.414674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:57.746488 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:57.746409    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" event={"ID":"faa828ae-d375-4578-9485-71ed468a41e1","Type":"ContainerStarted","Data":"bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02"}
Nov 03 01:01:57.811487 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:57.811459    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2612884-ed36-4883-88e7-b7e0ead7baa4" path="/var/lib/kubelet/pods/d2612884-ed36-4883-88e7-b7e0ead7baa4/volumes"
Nov 03 01:01:57.811965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:57.811952    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbca61a3-4d5b-4854-91c1-1765a16dd353" path="/var/lib/kubelet/pods/dbca61a3-4d5b-4854-91c1-1765a16dd353/volumes"
Nov 03 01:01:58.237364 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:58.237329    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 01:01:58.237676 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:58.237653    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 01:01:58.237732 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:58.237703    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 01:01:59.134760 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:59.134726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:01:59.238023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:59.237987    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 01:01:59.238212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:01:59.238085    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 01:02:00.237233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:00.237191    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 01:02:00.237672 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:00.237259    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 01:02:00.837950 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:00.837909    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:01.238267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:01.238236    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 01:02:01.238740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:01.238293    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 01:02:02.237411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:02.237374    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 01:02:02.237629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:02.237454    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 01:02:02.628177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:02.628091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:03.237355 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:03.237317    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body=
Nov 03 01:02:03.237550 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:03.237375    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused"
Nov 03 01:02:04.242957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:04.242928    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 01:02:04.243426 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:04.243312    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 01:02:04.243986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:04.243965    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 01:02:04.318323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:04.318284    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:05.045544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:05.045506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:05.938164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:05.938127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:07.705617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:07.705585    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:09.528911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:09.528873    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:10.033750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:10.033715    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"]
Nov 03 01:02:10.048015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:10.047978    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebe015c3-zm28z"]
Nov 03 01:02:10.054634 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:10.054598    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"]
Nov 03 01:02:10.058637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:10.058611    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad75d9ee-xhxg2"]
Nov 03 01:02:11.594152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:11.594101    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:11.825130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:11.825093    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6787d1d2-bf8f-4793-ab7e-dbc3964464f9" path="/var/lib/kubelet/pods/6787d1d2-bf8f-4793-ab7e-dbc3964464f9/volumes"
Nov 03 01:02:11.825665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:11.825650    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938de3a0-05f9-4c4f-8666-5f230d718a68" path="/var/lib/kubelet/pods/938de3a0-05f9-4c4f-8666-5f230d718a68/volumes"
Nov 03 01:02:13.441263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:13.441230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:14.400234 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:14.400199    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:15.321890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:15.321853    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:16.714088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:16.714050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:17.042990 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:17.042892    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:18.754445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:18.754393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:19.028868 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:19.028785    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"]
Nov 03 01:02:19.041607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:19.041574    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aacf581f-6q62v"]
Nov 03 01:02:19.810774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:19.810736    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babea784-bbcc-4f41-8284-9dcb22ab3539" path="/var/lib/kubelet/pods/babea784-bbcc-4f41-8284-9dcb22ab3539/volumes"
Nov 03 01:02:20.029842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:20.029810    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"]
Nov 03 01:02:20.042530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:20.042499    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37f890fc-5wwqb"]
Nov 03 01:02:20.434674 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:20.434640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:21.811837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:21.811792    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95ce9ef-83ec-42e9-ba3b-825d05c507cf" path="/var/lib/kubelet/pods/d95ce9ef-83ec-42e9-ba3b-825d05c507cf/volumes"
Nov 03 01:02:22.212371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:22.212340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:22.329170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:22.329133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:23.970402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:23.970369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:24.725700 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:02:24.725665    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:02:25.440476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.440389    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:25.684696 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.684660    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"]
Nov 03 01:02:25.695508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.686113    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:25.696793 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.696761    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:25.697121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697105    4550 cpu_manager.go:414] "containerMap: removing container" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.697224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697214    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.697310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697298    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy"
Nov 03 01:02:25.697376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697367    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy"
Nov 03 01:02:25.697491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697475    4550 cpu_manager.go:414] "containerMap: removing container" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:25.697610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697596    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:25.697725 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697713    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.697819 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697806    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.697915 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697904    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.698004 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.697992    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.698085 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698075    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:25.698172 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698161    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:25.698272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698260    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.698368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698355    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.698480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698469    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.698557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698548    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.698626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698618    4550 cpu_manager.go:414] "containerMap: removing container" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:25.698692 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698683    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:25.698770 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698761    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.698849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698838    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.698932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698922    4550 cpu_manager.go:414] "containerMap: removing container" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.699000 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.698992    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.699066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699057    4550 cpu_manager.go:414] "containerMap: removing container" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.699129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699122    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.699201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699193    4550 cpu_manager.go:414] "containerMap: removing container" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:25.699271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699263    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:25.699340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699331    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.699404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699395    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.699506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699496    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:25.699573 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699564    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:25.699641 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699633    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:25.699709 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699701    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:25.699774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699766    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.699846 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699837    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.699927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699919    4550 cpu_manager.go:414] "containerMap: removing container" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.699993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.699985    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.700061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.700054    4550 cpu_manager.go:414] "containerMap: removing container" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.700130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.700122    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.700202 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.700193    4550 cpu_manager.go:414] "containerMap: removing container" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.700272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.700263    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.700343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.700334    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:25.700411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.700401    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:25.725519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.725486    4550 cpu_manager.go:414] "containerMap: removing container" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:25.725772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.725755    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:25.725896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.725884    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.725993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.725982    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.726093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726084    4550 cpu_manager.go:414] "containerMap: removing container" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.726186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726177    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.726289 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726279    4550 cpu_manager.go:414] "containerMap: removing container" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:25.726486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726469    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:25.726611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726600    4550 cpu_manager.go:414] "containerMap: removing container" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.726706 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726695    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.726813 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726803    4550 cpu_manager.go:414] "containerMap: removing container" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.726908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.726897    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.727016 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727006    4550 cpu_manager.go:414] "containerMap: removing container" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:25.727127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727116    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:25.727235 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727226    4550 cpu_manager.go:414] "containerMap: removing container" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.727355 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727334    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.727482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727471    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.727640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727618    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.727758 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727748    4550 cpu_manager.go:414] "containerMap: removing container" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.727852 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727842    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.727946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.727937    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:25.728033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728024    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:25.728138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728128    4550 cpu_manager.go:414] "containerMap: removing container" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:25.728233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728220    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:25.728356 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728345    4550 cpu_manager.go:414] "containerMap: removing container" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.728478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728467    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.728583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728574    4550 cpu_manager.go:414] "containerMap: removing container" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.728672 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728662    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.728775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728763    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.728871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728861    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.728966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.728957    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:25.729054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.729044    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:25.729148 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.729139    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.729273 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.729251    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.729404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.729394    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.746691 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.746659    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.746881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.746869    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.746966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.746956    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.747054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747045    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:25.747129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747118    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:25.747223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747210    4550 cpu_manager.go:414] "containerMap: removing container" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:25.747310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747297    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:25.747409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747399    4550 cpu_manager.go:414] "containerMap: removing container" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.747526 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747512    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.747629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747618    4550 cpu_manager.go:414] "containerMap: removing container" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:25.747742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.747729    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:25.748484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.748464    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.748609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.748598    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.748698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.748689    4550 cpu_manager.go:414] "containerMap: removing container" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.748769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.748760    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.757977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.757953    4550 cpu_manager.go:414] "containerMap: removing container" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.758157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.758144    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.758255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.758243    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:25.761485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.761461    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:25.761657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.761645    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerName="payload-pre-processing"
Nov 03 01:02:25.761743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.761733    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerName="payload-pre-processing"
Nov 03 01:02:25.767527 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.767501    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.846179 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.846132    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.899744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.899715    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:25.899996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.899982    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:25.900103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900091    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:25.900184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900174    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:25.900269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900259    4550 cpu_manager.go:414] "containerMap: removing container" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:25.900353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900340    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:25.900456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900445    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.900565 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900554    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.900657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900645    4550 cpu_manager.go:414] "containerMap: removing container" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.900735 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900724    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.900826 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900815    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:25.900915 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.900904    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:25.901262 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.901240    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.907116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907095    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:25.907289 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907273    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:25.907401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907390    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:25.907557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907544    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.907660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907649    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.907753 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907743    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.907841 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907831    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:25.907936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.907926    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fb62e20-9899-476b-a1a4-0db41756bbd1" containerName="payload-pre-processing"
Nov 03 01:02:25.908021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908011    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:25.908105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908095    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:25.908187 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908177    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:25.908270 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908260    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:25.908348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908339    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.908480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908423    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:25.908582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908570    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:25.908684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908672    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:25.908775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908764    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4171a7bd-72e5-4863-9cdb-5637237dcddf" containerName="cleanup"
Nov 03 01:02:25.908866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908857    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:25.908983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.908971    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:25.909079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.909067    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:25.912476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.912454    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:25.913033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913016    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.913165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913147    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b34cfb-64e8-4c33-8a72-bc447bab068d" containerName="istio-proxy"
Nov 03 01:02:25.913165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913164    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913173    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913181    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913192    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913204    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913211    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913222    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913231    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913239    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e0a52de-9acc-4c3e-bb27-fa6137fee5df" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913248    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913256    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913264    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913278    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913286    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913298    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913307    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:25.913316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913316    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913326    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913336    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913347    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913355    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913367    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913377    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913388    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913397    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="03449032-8806-4415-a9ff-7ac5f37df4ab" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913406    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913416    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913462    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913472    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913493    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9598928-b49d-40ab-9e70-e127b122c329" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913503    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f332463-51f7-4d57-ae51-99f46e0074b6" containerName="cleanup"
Nov 03 01:02:25.914669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.913512    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:25.915802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.707945    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:25.924066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.923153    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:25.947990 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:25.938861    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"]
Nov 03 01:02:26.011229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011200    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011238    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011266    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxx4n\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011532    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011594    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011701 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011678    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011744    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011800    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.011882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.011864    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113365    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113411    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113479    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113527    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113575    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113604    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.113894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113869    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113908    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113947    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxx4n\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.113951    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114216 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.114173    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114334 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.114312    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114398 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.114338    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.114398 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.114358    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.115861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.115840    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.116512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.116494    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.133380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.133351    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.133495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.133417    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxx4n\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n\") pod \"maas-default-gateway-openshift-default-6cc458d887-mnd7q\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.257337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.257251    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:26.582669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.582639    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"]
Nov 03 01:02:26.583837 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:02:26.583812    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8374e40_3606_493e_9c09_f7ebecf7a6af.slice/crio-e65c08b409be039aca94194d4805abbd95684a2ce76bc06d7100191ef39d70f3 WatchSource:0}: Error finding container e65c08b409be039aca94194d4805abbd95684a2ce76bc06d7100191ef39d70f3: Status 404 returned error can't find the container with id e65c08b409be039aca94194d4805abbd95684a2ce76bc06d7100191ef39d70f3
Nov 03 01:02:26.586938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.586906    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:02:26.587058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.586973    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:02:26.587058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.587004    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:02:26.711821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.711782    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:26.812462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:26.812393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:27.084571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.084526    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" event={"ID":"f8374e40-3606-493e-9c09-f7ebecf7a6af","Type":"ContainerStarted","Data":"590cb62a9db3252f6d0eba46a77b3ea7bf76a71de2c834907bd6464df6275b76"}
Nov 03 01:02:27.084774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.084641    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" event={"ID":"f8374e40-3606-493e-9c09-f7ebecf7a6af","Type":"ContainerStarted","Data":"e65c08b409be039aca94194d4805abbd95684a2ce76bc06d7100191ef39d70f3"}
Nov 03 01:02:27.128570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.128518    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podStartSLOduration=2.128503621 podStartE2EDuration="2.128503621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:27.127528242 +0000 UTC m=+3429.449065062" watchObservedRunningTime="2026-11-03 01:02:27.128503621 +0000 UTC m=+3429.450040421"
Nov 03 01:02:27.257818 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.257768    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:27.259888 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.259858    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:27.260044 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.259909    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:27.440639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:27.440607    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:28.264457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:28.260845    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:28.264457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:28.260924    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:29.222995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:29.222958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:29.262457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:29.259873    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:29.262457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:29.259955    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:30.259929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:30.259886    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:30.260596 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:30.260560    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:31.091542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:31.091505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:31.276471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:31.275966    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:31.276471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:31.276048    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:32.268890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:32.268846    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:32.269212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:32.269181    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:32.833626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:32.833591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:33.258573 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:33.258526    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:33.258809 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:33.258616    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:34.257877 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:34.257828    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-mnd7q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body=
Nov 03 01:02:34.258419 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:34.257923    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused"
Nov 03 01:02:34.652577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:34.652539    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:35.959732 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:35.959696    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:35.961263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:35.961230    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:35.961893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:35.961871    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:02:36.351001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:36.350910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:38.163449 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:38.163393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:39.933197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:39.933152    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:41.745420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:41.745378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:43.414938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:43.414905    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:45.049701 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.049657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:45.630695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.630659    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"]
Nov 03 01:02:45.631386 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631372    4550 cpu_manager.go:414] "containerMap: removing container" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631387    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631405    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631410    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631442    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631447    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631454    4550 cpu_manager.go:414] "containerMap: removing container" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631459    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631470    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631475    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:45.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631485    4550 cpu_manager.go:414] "containerMap: removing container" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631490    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540463f3-b44b-4da2-adeb-192f817fcf3c" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631497    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631502    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631509    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631514    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631522    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631527    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd14ecf-d850-4d11-9951-22c62bea5ffb" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631542    4550 cpu_manager.go:414] "containerMap: removing container" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631546    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c9afba-79f5-41d3-ae1e-0cf85863e6d6" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631554    4550 cpu_manager.go:414] "containerMap: removing container" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631559    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631576    4550 cpu_manager.go:414] "containerMap: removing container" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631580    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631587    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631591    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631605    4550 cpu_manager.go:414] "containerMap: removing container" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631609    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631720    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70efcea-af98-4d4a-bd08-490cb86f055b" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631729    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca04320-6331-4640-b213-c761707a2024" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631740    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a5ddfa-213d-472a-9460-3e770c9cb1a8" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631746    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e9e7b8-65ef-47db-bb3e-8a62be6eef4c" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631753    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa394189-738e-4dc7-8cef-c0576e53a98d" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631763    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="deec6453-23fd-49d0-aded-4518c0e1b5e2" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631770    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a4e5e2-d23b-4006-a3fa-448ded9b054f" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631777    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="90507494-4818-4eb4-a1cb-a3e682b3bdd8" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631783    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53c2794-70d3-497e-91f6-08e4ec16bf59" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631790    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3442fb4-b070-4d02-b0a7-0f78e83f5c61" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631798    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1c2e9f-ea32-4606-9d45-a0aa3ad187ce" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631805    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="620eee4a-a44a-45ce-8cdc-b9a4368aa6ab" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631814    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="068293c7-3cf2-4d37-94c7-958ca42a55f5" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631821    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd75e99a-1821-4bb0-8f2e-16ce5c44db28" containerName="cleanup"
Nov 03 01:02:45.631861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.631827    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="3117c03f-2cc4-467a-8716-fb4e1b6c0f5e" containerName="cleanup"
Nov 03 01:02:45.635943 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.635924    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.654506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.654466    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"]
Nov 03 01:02:45.660860 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.660830    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:02:45.661030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.660917    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:02:45.686344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.686286    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.686590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.686393    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhlh\" (UniqueName: \"kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.686590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.686503    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.787933 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.787895    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.788181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.787972    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhlh\" (UniqueName: \"kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.788181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.788037    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.788775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.788744    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.790553 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.790529    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.808267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.808234    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhlh\" (UniqueName: \"kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh\") pod \"maas-api-revoke-sub-3a0feefd-hrzds\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:45.953922 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:45.953885    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:46.261102 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:02:46.261062    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d68cef5_adeb_4b15_80f8_6828b5caba42.slice/crio-661a4427f044e6e0b2a4ba4e735a4fad3e8e21d84bae08873b3a5af76b49b1bf WatchSource:0}: Error finding container 661a4427f044e6e0b2a4ba4e735a4fad3e8e21d84bae08873b3a5af76b49b1bf: Status 404 returned error can't find the container with id 661a4427f044e6e0b2a4ba4e735a4fad3e8e21d84bae08873b3a5af76b49b1bf
Nov 03 01:02:46.262944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:46.262913    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"]
Nov 03 01:02:46.728714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:46.728663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:47.268994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:47.268951    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds" event={"ID":"6d68cef5-adeb-4b15-80f8-6828b5caba42","Type":"ContainerStarted","Data":"1dc6af3f4347a8c96a3712f36f83f06b8c6a054d2934db4db97ffd699a503ed0"}
Nov 03 01:02:47.269534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:47.269023    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds" event={"ID":"6d68cef5-adeb-4b15-80f8-6828b5caba42","Type":"ContainerStarted","Data":"661a4427f044e6e0b2a4ba4e735a4fad3e8e21d84bae08873b3a5af76b49b1bf"}
Nov 03 01:02:47.314075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:47.314018    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds" podStartSLOduration=2.314004789 podStartE2EDuration="2.314004789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:47.310177577 +0000 UTC m=+3449.631714377" watchObservedRunningTime="2026-11-03 01:02:47.314004789 +0000 UTC m=+3449.635541590"
Nov 03 01:02:48.282860 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:48.282816    4550 generic.go:356] "Generic (PLEG): container finished" podID="6d68cef5-adeb-4b15-80f8-6828b5caba42" containerID="1dc6af3f4347a8c96a3712f36f83f06b8c6a054d2934db4db97ffd699a503ed0" exitCode=0
Nov 03 01:02:48.283309 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:48.282903    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds" event={"ID":"6d68cef5-adeb-4b15-80f8-6828b5caba42","Type":"ContainerDied","Data":"1dc6af3f4347a8c96a3712f36f83f06b8c6a054d2934db4db97ffd699a503ed0"}
Nov 03 01:02:48.416607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:48.416569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:49.492177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.492143    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:49.546919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.546822    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca\") pod \"6d68cef5-adeb-4b15-80f8-6828b5caba42\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") "
Nov 03 01:02:49.546919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.546901    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh\" (UniqueName: \"kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh\") pod \"6d68cef5-adeb-4b15-80f8-6828b5caba42\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") "
Nov 03 01:02:49.547159 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.547091    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca\") pod \"6d68cef5-adeb-4b15-80f8-6828b5caba42\" (UID: \"6d68cef5-adeb-4b15-80f8-6828b5caba42\") "
Nov 03 01:02:49.547759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.547725    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca" pod "6d68cef5-adeb-4b15-80f8-6828b5caba42" (UID: "6d68cef5-adeb-4b15-80f8-6828b5caba42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:02:49.547923 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.547896    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:49.549808 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.549779    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca" pod "6d68cef5-adeb-4b15-80f8-6828b5caba42" (UID: "6d68cef5-adeb-4b15-80f8-6828b5caba42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:02:49.549992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.549971    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh" pod "6d68cef5-adeb-4b15-80f8-6828b5caba42" (UID: "6d68cef5-adeb-4b15-80f8-6828b5caba42"). InnerVolumeSpecName "kube-api-access-jdhlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:02:49.649543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.649509    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d68cef5-adeb-4b15-80f8-6828b5caba42-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:49.649543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:49.649537    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhlh\" (UniqueName: \"kubernetes.io/projected/6d68cef5-adeb-4b15-80f8-6828b5caba42-kube-api-access-jdhlh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:50.005828 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:50.005778    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:50.306265 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:50.306238    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"
Nov 03 01:02:50.306482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:50.306226    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds" event={"ID":"6d68cef5-adeb-4b15-80f8-6828b5caba42","Type":"ContainerDied","Data":"661a4427f044e6e0b2a4ba4e735a4fad3e8e21d84bae08873b3a5af76b49b1bf"}
Nov 03 01:02:50.306482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:50.306364    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="661a4427f044e6e0b2a4ba4e735a4fad3e8e21d84bae08873b3a5af76b49b1bf"
Nov 03 01:02:51.741130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:51.741093    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:53.400492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:53.400403    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:54.727227 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:02:54.727182    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:02:55.061007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:55.060925    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:55.864929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:55.864889    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"]
Nov 03 01:02:55.865505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:55.865103    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" containerName="discovery" containerID="cri-o://6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997" gracePeriod=30
Nov 03 01:02:55.925958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:55.925913    4550 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kzsj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.219:8080/ready\": dial tcp 10.133.0.219:8080: connect: connection refused" start-of-body=
Nov 03 01:02:55.926396 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:55.925989    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.219:8080/ready\": dial tcp 10.133.0.219:8080: connect: connection refused"
Nov 03 01:02:56.049806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.049768    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:56.180016 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.179983    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 01:02:56.256245 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256201    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.256468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256267    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.256468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256299    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.256619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256483    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.256619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256569    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.256743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256722    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.256803 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256765    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:02:56.256803 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.256784    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig\") pod \"8578b459-8489-48d7-a768-85e55cfaef3d\" (UID: \"8578b459-8489-48d7-a768-85e55cfaef3d\") "
Nov 03 01:02:56.258772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.258662    4550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-ca-configmap\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.260668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.260526    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:02:56.260668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.260640    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:02:56.278486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.270952    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:02:56.278486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.270955    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:02:56.278486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.271046    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:02:56.278486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.271083    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq" pod "8578b459-8489-48d7-a768-85e55cfaef3d" (UID: "8578b459-8489-48d7-a768-85e55cfaef3d"). InnerVolumeSpecName "kube-api-access-5v9rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:02:56.357121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.357081    4550 generic.go:356] "Generic (PLEG): container finished" podID="8578b459-8489-48d7-a768-85e55cfaef3d" containerID="6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997" exitCode=0
Nov 03 01:02:56.357350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.357142    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"
Nov 03 01:02:56.357350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.357167    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" event={"ID":"8578b459-8489-48d7-a768-85e55cfaef3d","Type":"ContainerDied","Data":"6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997"}
Nov 03 01:02:56.357350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.357227    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7" event={"ID":"8578b459-8489-48d7-a768-85e55cfaef3d","Type":"ContainerDied","Data":"ff43abfbaf8299f482f121138c703d9a19e417669e338086333de47080e3e42b"}
Nov 03 01:02:56.357350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.357256    4550 scope.go:122] "RemoveContainer" containerID="6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997"
Nov 03 01:02:56.359971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.359947    4550 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-csr-dns-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.359971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.359971    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.360182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.359981    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v9rq\" (UniqueName: \"kubernetes.io/projected/8578b459-8489-48d7-a768-85e55cfaef3d-kube-api-access-5v9rq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.360182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.359992    4550 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-istio-kubeconfig\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.360182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.360000    4550 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8578b459-8489-48d7-a768-85e55cfaef3d-local-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.360182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.360008    4550 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8578b459-8489-48d7-a768-85e55cfaef3d-cacerts\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:02:56.369205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.369187    4550 scope.go:122] "RemoveContainer" containerID="6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997"
Nov 03 01:02:56.369651 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:02:56.369624    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997\": container with ID starting with 6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997 not found: ID does not exist" containerID="6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997"
Nov 03 01:02:56.369732 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.369666    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997"} err="failed to get container status \"6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997\": rpc error: code = NotFound desc = could not find container \"6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997\": container with ID starting with 6b269dfba75731d87047079a10354765396ed4d226527a433b6cbc2dfa8c4997 not found: ID does not exist"
Nov 03 01:02:56.417539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.417510    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"]
Nov 03 01:02:56.422251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.422222    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kzsj7"]
Nov 03 01:02:56.753528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:56.753384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:02:57.815027 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:57.814994    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" path="/var/lib/kubelet/pods/8578b459-8489-48d7-a768-85e55cfaef3d/volumes"
Nov 03 01:02:58.450873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:02:58.450825    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:00.129915 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:00.129824    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:01.593426 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:01.593385    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"]
Nov 03 01:03:01.593995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:01.593621    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" containerID="cri-o://9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b" gracePeriod=30
Nov 03 01:03:01.841068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:01.841026    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:02.442193 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:02.442147    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:03.532273 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:03.532225    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:05.222158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:05.222118    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:06.830862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:06.830776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:06.940058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:06.940025    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 01:03:07.039707 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039663    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.039924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039726    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.039924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039790    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.039924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039833    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.039924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039868    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.040170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039950    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.040170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.039982    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:07.040286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040248    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.040345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040288    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.040400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040345    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert\") pod \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\" (UID: \"2b6cb3db-89da-44c7-b194-5150ec82a1d8\") "
Nov 03 01:03:07.040524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040498    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:07.040617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040600    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:07.040883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040850    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:07.041034 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.040968    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:07.041245 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.041223    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.041245 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.041244    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.041414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.041254    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.041414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.041264    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.041414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.041272    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.043021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.042993    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:03:07.043157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.043138    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:07.043756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.043732    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "kube-api-access-vwb6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:07.044680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.044661    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy" pod "2b6cb3db-89da-44c7-b194-5150ec82a1d8" (UID: "2b6cb3db-89da-44c7-b194-5150ec82a1d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:07.142495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.142390    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.142495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.142422    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwb6v\" (UniqueName: \"kubernetes.io/projected/2b6cb3db-89da-44c7-b194-5150ec82a1d8-kube-api-access-vwb6v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.142495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.142442    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.142495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.142450    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b6cb3db-89da-44c7-b194-5150ec82a1d8-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:07.451328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.451288    4550 generic.go:356] "Generic (PLEG): container finished" podID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerID="9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b" exitCode=0
Nov 03 01:03:07.451562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.451349    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"
Nov 03 01:03:07.451562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.451372    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" event={"ID":"2b6cb3db-89da-44c7-b194-5150ec82a1d8","Type":"ContainerDied","Data":"9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b"}
Nov 03 01:03:07.451562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.451444    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" event={"ID":"2b6cb3db-89da-44c7-b194-5150ec82a1d8","Type":"ContainerDied","Data":"752f0a1b33cc06a2acec09eda925d23dd2f973db3efaa2ef7c5012fd80e14750"}
Nov 03 01:03:07.451562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.451467    4550 scope.go:122] "RemoveContainer" containerID="9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b"
Nov 03 01:03:07.465243 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.465223    4550 scope.go:122] "RemoveContainer" containerID="9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b"
Nov 03 01:03:07.465617 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:03:07.465591    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b\": container with ID starting with 9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b not found: ID does not exist" containerID="9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b"
Nov 03 01:03:07.465678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.465631    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b"} err="failed to get container status \"9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b\": rpc error: code = NotFound desc = could not find container \"9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b\": container with ID starting with 9c0df537b188b16a0f551cfdceadc30b3ebcf3a46426e56c61b9183cf31e586b not found: ID does not exist"
Nov 03 01:03:07.516018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.515934    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"]
Nov 03 01:03:07.522698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.522669    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw"]
Nov 03 01:03:07.809952 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.809874    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" path="/var/lib/kubelet/pods/2b6cb3db-89da-44c7-b194-5150ec82a1d8/volumes"
Nov 03 01:03:07.929631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.929580    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:03:07.930069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:07.929664    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-jrljw" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:03:08.435074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:08.435039    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:10.087677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:10.087634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:11.771845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:11.771808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:13.459653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:13.459615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:14.439167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:14.439132    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:15.211490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:15.211427    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:16.942737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:16.942700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:18.538403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:18.538365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:20.540165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:20.540127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:21.172657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.172619    4550 scope.go:122] "RemoveContainer" containerID="c25d9923d752902e8b290087e8d4467a3f6841b206399c4317990f0306278a44"
Nov 03 01:03:21.255561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.255419    4550 scope.go:122] "RemoveContainer" containerID="8cbd40f878c049c7545909a04cddf68ce294191d52cf90609847e5b468eabd6f"
Nov 03 01:03:21.358876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.358846    4550 scope.go:122] "RemoveContainer" containerID="0375bd5af04f02265195fb22c3a074a24ada1419cc86a8295294b3a7f2900958"
Nov 03 01:03:21.450657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.450624    4550 scope.go:122] "RemoveContainer" containerID="54933def303535fd0ac260c924a0e52fb045d971b8bd5b297e4c3b2aae726dd9"
Nov 03 01:03:21.559966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.559922    4550 scope.go:122] "RemoveContainer" containerID="3c2bc97f627ec6d1998698a21f2d0ec2cb848a1df2eccbea3b6a52f6aa32d36b"
Nov 03 01:03:21.715036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.675448    4550 scope.go:122] "RemoveContainer" containerID="a8b6d1d92852aa8e0a8f4deee59edfa036582fc13c5d17d6565efde604266492"
Nov 03 01:03:21.754455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.752129    4550 scope.go:122] "RemoveContainer" containerID="cc4e997a2cffe28216c4d77d3ae685c272a1ea34ecfc928b49c8d1f5a5407233"
Nov 03 01:03:21.990864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:21.990834    4550 scope.go:122] "RemoveContainer" containerID="120f3c56a234db383ebff7ace076864595262709059628572f3407bcf8ccd182"
Nov 03 01:03:23.031870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:23.031838    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:24.725809 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:03:24.725768    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:03:24.727211 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:24.727168    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:26.383208 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:26.383169    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:28.123058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:28.123023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:29.755054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:29.755019    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:31.335369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:31.335323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:33.025058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:33.025022    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:34.802336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:34.802291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:36.439678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:36.439639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:38.041512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:38.041472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:39.797140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:39.797086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:41.485153 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:41.485111    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:43.133192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:43.133150    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:44.838217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838181    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"]
Nov 03 01:03:44.838924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838909    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6d68cef5-adeb-4b15-80f8-6828b5caba42" containerName="revoke-keys"
Nov 03 01:03:44.838970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838925    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d68cef5-adeb-4b15-80f8-6828b5caba42" containerName="revoke-keys"
Nov 03 01:03:44.838970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838958    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy"
Nov 03 01:03:44.838970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838964    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy"
Nov 03 01:03:44.839075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838988    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" containerName="discovery"
Nov 03 01:03:44.839075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.838993    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" containerName="discovery"
Nov 03 01:03:44.839145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.839090    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d68cef5-adeb-4b15-80f8-6828b5caba42" containerName="revoke-keys"
Nov 03 01:03:44.839145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.839102    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b6cb3db-89da-44c7-b194-5150ec82a1d8" containerName="istio-proxy"
Nov 03 01:03:44.839145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.839111    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8578b459-8489-48d7-a768-85e55cfaef3d" containerName="discovery"
Nov 03 01:03:44.844764 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.844744    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.861004 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.860976    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"]
Nov 03 01:03:44.875778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.875746    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:03:44.875778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.875757    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:03:44.893313 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.893281    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.893533 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.893425    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt7h6\" (UniqueName: \"kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.893612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.893565    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.914787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.914710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:44.996711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.996667    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt7h6\" (UniqueName: \"kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.997183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.997158    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.997539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.997517    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:44.998524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:44.998500    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:45.011971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:45.011942    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:45.026879 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:45.026223    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt7h6\" (UniqueName: \"kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6\") pod \"maas-api-revoke-sub-5359e2a7-k6djn\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:45.159401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:45.159346    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:45.469872 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:03:45.469819    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ce451b_db6e_4107_9842_514f4b20328b.slice/crio-c7de2a63fcb4e342c2070005d4a26814e8aca3a2d1c5e2b0f8961b78e21243d7 WatchSource:0}: Error finding container c7de2a63fcb4e342c2070005d4a26814e8aca3a2d1c5e2b0f8961b78e21243d7: Status 404 returned error can't find the container with id c7de2a63fcb4e342c2070005d4a26814e8aca3a2d1c5e2b0f8961b78e21243d7
Nov 03 01:03:45.470054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:45.469937    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"]
Nov 03 01:03:46.149312 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:46.149217    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn" event={"ID":"92ce451b-db6e-4107-9842-514f4b20328b","Type":"ContainerStarted","Data":"b530735616e0e10cd83e8204190e723d4e120d258bf2a2bbe48a9ed91c89aa04"}
Nov 03 01:03:46.149312 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:46.149288    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn" event={"ID":"92ce451b-db6e-4107-9842-514f4b20328b","Type":"ContainerStarted","Data":"c7de2a63fcb4e342c2070005d4a26814e8aca3a2d1c5e2b0f8961b78e21243d7"}
Nov 03 01:03:46.192449 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:46.192375    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn" podStartSLOduration=2.192361275 podStartE2EDuration="2.192361275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:46.188495266 +0000 UTC m=+3508.510032067" watchObservedRunningTime="2026-11-03 01:03:46.192361275 +0000 UTC m=+3508.513898098"
Nov 03 01:03:46.517014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:46.516980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:47.158807 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:47.158773    4550 generic.go:356] "Generic (PLEG): container finished" podID="92ce451b-db6e-4107-9842-514f4b20328b" containerID="b530735616e0e10cd83e8204190e723d4e120d258bf2a2bbe48a9ed91c89aa04" exitCode=0
Nov 03 01:03:47.159261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:47.158841    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn" event={"ID":"92ce451b-db6e-4107-9842-514f4b20328b","Type":"ContainerDied","Data":"b530735616e0e10cd83e8204190e723d4e120d258bf2a2bbe48a9ed91c89aa04"}
Nov 03 01:03:48.209819 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.209771    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:48.409752 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.409722    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:48.451610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.451518    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca\") pod \"92ce451b-db6e-4107-9842-514f4b20328b\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") "
Nov 03 01:03:48.451874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.451854    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6\" (UniqueName: \"kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6\") pod \"92ce451b-db6e-4107-9842-514f4b20328b\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") "
Nov 03 01:03:48.452068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.451920    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca\") pod \"92ce451b-db6e-4107-9842-514f4b20328b\" (UID: \"92ce451b-db6e-4107-9842-514f4b20328b\") "
Nov 03 01:03:48.452666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.452622    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca" pod "92ce451b-db6e-4107-9842-514f4b20328b" (UID: "92ce451b-db6e-4107-9842-514f4b20328b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:48.455173 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.455138    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6" pod "92ce451b-db6e-4107-9842-514f4b20328b" (UID: "92ce451b-db6e-4107-9842-514f4b20328b"). InnerVolumeSpecName "kube-api-access-gt7h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:48.455840 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.455809    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca" pod "92ce451b-db6e-4107-9842-514f4b20328b" (UID: "92ce451b-db6e-4107-9842-514f4b20328b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:03:48.554022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.553986    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt7h6\" (UniqueName: \"kubernetes.io/projected/92ce451b-db6e-4107-9842-514f4b20328b-kube-api-access-gt7h6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:48.554022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.554022    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:48.554298 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:48.554038    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce451b-db6e-4107-9842-514f4b20328b-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:49.200654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:49.182608    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn" event={"ID":"92ce451b-db6e-4107-9842-514f4b20328b","Type":"ContainerDied","Data":"c7de2a63fcb4e342c2070005d4a26814e8aca3a2d1c5e2b0f8961b78e21243d7"}
Nov 03 01:03:49.200654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:49.182715    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7de2a63fcb4e342c2070005d4a26814e8aca3a2d1c5e2b0f8961b78e21243d7"
Nov 03 01:03:49.200654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:49.182802    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"
Nov 03 01:03:50.001804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:50.001767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:51.574967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:51.574919    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:53.305411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:53.305370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:54.725225 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:03:54.725188    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:03:54.938279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:54.938242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:55.807042 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:55.806985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:56.399096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.399055    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"]
Nov 03 01:03:56.399884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.399864    4550 cpu_manager.go:414] "containerMap: removing container" podUID="92ce451b-db6e-4107-9842-514f4b20328b" containerName="revoke-keys"
Nov 03 01:03:56.399884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.399886    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ce451b-db6e-4107-9842-514f4b20328b" containerName="revoke-keys"
Nov 03 01:03:56.400043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.400034    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ce451b-db6e-4107-9842-514f4b20328b" containerName="revoke-keys"
Nov 03 01:03:56.403778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.403757    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.423269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.423241    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"]
Nov 03 01:03:56.430099 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.430071    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:03:56.430379 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.430335    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:03:56.477288 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.477246    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlqh\" (UniqueName: \"kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.477526 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.477407    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.477526 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.477506    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.579084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.579044    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlqh\" (UniqueName: \"kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.579285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.579149    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.579285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.579219    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.580003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.579976    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.581823 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.581797    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.600048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.600016    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlqh\" (UniqueName: \"kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh\") pod \"maas-api-revoke-sub-cc13d928-kkkpx\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.718601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.718568    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:56.828115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:56.828076    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:57.026667 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:03:57.026630    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07b257b_661d_4b2d_8d11_47cfc653efd3.slice/crio-59a98b5e11c5f5f9d6f9a743619d129127cde6332ad6cebfed9af9fed9422b2e WatchSource:0}: Error finding container 59a98b5e11c5f5f9d6f9a743619d129127cde6332ad6cebfed9af9fed9422b2e: Status 404 returned error can't find the container with id 59a98b5e11c5f5f9d6f9a743619d129127cde6332ad6cebfed9af9fed9422b2e
Nov 03 01:03:57.028215 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:57.028189    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"]
Nov 03 01:03:57.244426 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:57.244387    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx" event={"ID":"c07b257b-661d-4b2d-8d11-47cfc653efd3","Type":"ContainerStarted","Data":"59a98b5e11c5f5f9d6f9a743619d129127cde6332ad6cebfed9af9fed9422b2e"}
Nov 03 01:03:58.253876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:58.253833    4550 generic.go:356] "Generic (PLEG): container finished" podID="c07b257b-661d-4b2d-8d11-47cfc653efd3" containerID="7d85ca07d1154cdd293f8269c3a4437943b463e31cff6383d2ce7c2465b2b317" exitCode=0
Nov 03 01:03:58.254301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:58.253913    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx" event={"ID":"c07b257b-661d-4b2d-8d11-47cfc653efd3","Type":"ContainerDied","Data":"7d85ca07d1154cdd293f8269c3a4437943b463e31cff6383d2ce7c2465b2b317"}
Nov 03 01:03:58.430041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:58.429935    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:03:59.444363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.444336    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:03:59.543381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.543290    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh\" (UniqueName: \"kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh\") pod \"c07b257b-661d-4b2d-8d11-47cfc653efd3\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") "
Nov 03 01:03:59.543621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.543537    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca\") pod \"c07b257b-661d-4b2d-8d11-47cfc653efd3\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") "
Nov 03 01:03:59.543706 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.543689    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca\") pod \"c07b257b-661d-4b2d-8d11-47cfc653efd3\" (UID: \"c07b257b-661d-4b2d-8d11-47cfc653efd3\") "
Nov 03 01:03:59.544125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.544094    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca" pod "c07b257b-661d-4b2d-8d11-47cfc653efd3" (UID: "c07b257b-661d-4b2d-8d11-47cfc653efd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:59.544445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.544406    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:59.551935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.551907    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca" pod "c07b257b-661d-4b2d-8d11-47cfc653efd3" (UID: "c07b257b-661d-4b2d-8d11-47cfc653efd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:03:59.552060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.551975    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh" pod "c07b257b-661d-4b2d-8d11-47cfc653efd3" (UID: "c07b257b-661d-4b2d-8d11-47cfc653efd3"). InnerVolumeSpecName "kube-api-access-lmlqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:59.646268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.646235    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c07b257b-661d-4b2d-8d11-47cfc653efd3-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:03:59.646268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:03:59.646265    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlqh\" (UniqueName: \"kubernetes.io/projected/c07b257b-661d-4b2d-8d11-47cfc653efd3-kube-api-access-lmlqh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:00.055995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:00.055953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:00.274211 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:00.274177    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"
Nov 03 01:04:00.274446 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:00.274244    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx" event={"ID":"c07b257b-661d-4b2d-8d11-47cfc653efd3","Type":"ContainerDied","Data":"59a98b5e11c5f5f9d6f9a743619d129127cde6332ad6cebfed9af9fed9422b2e"}
Nov 03 01:04:00.274446 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:00.274337    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59a98b5e11c5f5f9d6f9a743619d129127cde6332ad6cebfed9af9fed9422b2e"
Nov 03 01:04:01.830951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:01.830916    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:03.545265 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:03.545235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:05.145897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:05.145859    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:06.798057 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:06.798014    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:08.469615 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:08.469577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:10.204421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:10.204379    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:11.914701 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:11.914667    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:12.480299 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.480266    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"]
Nov 03 01:04:12.481019 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.481002    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c07b257b-661d-4b2d-8d11-47cfc653efd3" containerName="revoke-keys"
Nov 03 01:04:12.481019 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.481019    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07b257b-661d-4b2d-8d11-47cfc653efd3" containerName="revoke-keys"
Nov 03 01:04:12.481143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.481134    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07b257b-661d-4b2d-8d11-47cfc653efd3" containerName="revoke-keys"
Nov 03 01:04:12.484996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.484980    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.505658 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.505625    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:04:12.505789 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.505711    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:04:12.523607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.523577    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"]
Nov 03 01:04:12.567147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.567050    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.567147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.567115    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.567426 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.567401    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnxv2\" (UniqueName: \"kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.668876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.668842    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnxv2\" (UniqueName: \"kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.669073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.669024    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.669073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.669051    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.669898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.669869    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.671659 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.671636    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.689009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.688977    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnxv2\" (UniqueName: \"kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2\") pod \"maas-api-revoke-sub-2e668813-vq5x4\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:12.798812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:12.798718    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:13.103087 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:04:13.103054    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7320a705_f3f3_4aa3_af4a_15972677bc0b.slice/crio-b1ba0075c334769d80f3a9e2ca5e90b83c074217c3cc139485461a1dad9c006f WatchSource:0}: Error finding container b1ba0075c334769d80f3a9e2ca5e90b83c074217c3cc139485461a1dad9c006f: Status 404 returned error can't find the container with id b1ba0075c334769d80f3a9e2ca5e90b83c074217c3cc139485461a1dad9c006f
Nov 03 01:04:13.105996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:13.105957    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"]
Nov 03 01:04:13.396234 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:13.396138    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4" event={"ID":"7320a705-f3f3-4aa3-af4a-15972677bc0b","Type":"ContainerStarted","Data":"b1ba0075c334769d80f3a9e2ca5e90b83c074217c3cc139485461a1dad9c006f"}
Nov 03 01:04:13.650158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:13.650121    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:13.752010 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:13.751953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:14.408632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:14.408599    4550 generic.go:356] "Generic (PLEG): container finished" podID="7320a705-f3f3-4aa3-af4a-15972677bc0b" containerID="fe44533c5a9389f2bc4c328930ca4d0a9cbd164d00f32b74ab44b06c9b6b6185" exitCode=0
Nov 03 01:04:14.409032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:14.408680    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4" event={"ID":"7320a705-f3f3-4aa3-af4a-15972677bc0b","Type":"ContainerDied","Data":"fe44533c5a9389f2bc4c328930ca4d0a9cbd164d00f32b74ab44b06c9b6b6185"}
Nov 03 01:04:14.559467 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:14.559373    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:15.336264 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.336228    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:15.633833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.633804    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:15.731944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.731904    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca\") pod \"7320a705-f3f3-4aa3-af4a-15972677bc0b\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") "
Nov 03 01:04:15.732136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.732045    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2\" (UniqueName: \"kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2\") pod \"7320a705-f3f3-4aa3-af4a-15972677bc0b\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") "
Nov 03 01:04:15.732136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.732110    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca\") pod \"7320a705-f3f3-4aa3-af4a-15972677bc0b\" (UID: \"7320a705-f3f3-4aa3-af4a-15972677bc0b\") "
Nov 03 01:04:15.732559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.732529    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca" pod "7320a705-f3f3-4aa3-af4a-15972677bc0b" (UID: "7320a705-f3f3-4aa3-af4a-15972677bc0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:15.732949 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.732921    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:15.734324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.734283    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca" pod "7320a705-f3f3-4aa3-af4a-15972677bc0b" (UID: "7320a705-f3f3-4aa3-af4a-15972677bc0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:15.735075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.735047    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2" pod "7320a705-f3f3-4aa3-af4a-15972677bc0b" (UID: "7320a705-f3f3-4aa3-af4a-15972677bc0b"). InnerVolumeSpecName "kube-api-access-cnxv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:15.835121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.835088    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnxv2\" (UniqueName: \"kubernetes.io/projected/7320a705-f3f3-4aa3-af4a-15972677bc0b-kube-api-access-cnxv2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:15.835121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:15.835121    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7320a705-f3f3-4aa3-af4a-15972677bc0b-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:16.450502 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:16.450470    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"
Nov 03 01:04:16.450697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:16.450464    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4" event={"ID":"7320a705-f3f3-4aa3-af4a-15972677bc0b","Type":"ContainerDied","Data":"b1ba0075c334769d80f3a9e2ca5e90b83c074217c3cc139485461a1dad9c006f"}
Nov 03 01:04:16.450697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:16.450594    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ba0075c334769d80f3a9e2ca5e90b83c074217c3cc139485461a1dad9c006f"
Nov 03 01:04:16.959630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:16.959593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:18.648974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:18.648925    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:18.749666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:18.749631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:19.120570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:19.120529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:20.442566 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:20.442529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:22.043954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:22.043917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:23.760139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:23.760104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:24.701722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:24.701652    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:24.742057 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:04:24.742015    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:04:24.832457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:24.826499    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:25.224962 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:25.224926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:25.670806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:25.670772    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:26.578451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:26.578405    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:26.684901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:26.684861    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:26.784394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:26.784354    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:27.400230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:27.400201    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:27.500810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:27.500765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:29.008474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:29.008442    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:30.826703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:30.826604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:30.927213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:30.927178    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:32.582962 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:32.582922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:34.376968 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:34.376934    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:36.243881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:36.243843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:38.028957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:38.028921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:39.722274 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:39.722236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:40.957658 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:40.957622    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"]
Nov 03 01:04:40.999835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:40.999636    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:41.470466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.470392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:41.492768 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.492732    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"]
Nov 03 01:04:41.493552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.493534    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7320a705-f3f3-4aa3-af4a-15972677bc0b" containerName="revoke-keys"
Nov 03 01:04:41.493552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.493552    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7320a705-f3f3-4aa3-af4a-15972677bc0b" containerName="revoke-keys"
Nov 03 01:04:41.493671 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.493662    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7320a705-f3f3-4aa3-af4a-15972677bc0b" containerName="revoke-keys"
Nov 03 01:04:41.497469 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.497447    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.512711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.512682    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:04:41.514793 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.514765    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"]
Nov 03 01:04:41.523061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.523027    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:04:41.698972 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.698932    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwmt\" (UniqueName: \"kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.699216 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.699022    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.699216 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.699107    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.801393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.801292    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwmt\" (UniqueName: \"kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.801630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.801399    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.801630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.801463    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.802259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.802227    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.804165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.804140    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:41.822034 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:41.822008    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwmt\" (UniqueName: \"kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt\") pod \"maas-api-revoke-sub-fe6f97b5-7lrj8\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:42.113987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:42.113902    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:42.425350 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:04:42.425313    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04bba05_aee0_497f_b36f_4f91659da630.slice/crio-ca280416b2d551e5c3d8e33a344982b0ecf5ae4fd1e3b01bef184ff15c346aa9 WatchSource:0}: Error finding container ca280416b2d551e5c3d8e33a344982b0ecf5ae4fd1e3b01bef184ff15c346aa9: Status 404 returned error can't find the container with id ca280416b2d551e5c3d8e33a344982b0ecf5ae4fd1e3b01bef184ff15c346aa9
Nov 03 01:04:42.427614 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:42.427582    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"]
Nov 03 01:04:42.678351 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:42.678229    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8" event={"ID":"c04bba05-aee0-497f-b36f-4f91659da630","Type":"ContainerStarted","Data":"ca280416b2d551e5c3d8e33a344982b0ecf5ae4fd1e3b01bef184ff15c346aa9"}
Nov 03 01:04:43.134620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:43.134519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:43.688784 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:43.688747    4550 generic.go:356] "Generic (PLEG): container finished" podID="c04bba05-aee0-497f-b36f-4f91659da630" containerID="4142e6f84257b8948a3ff1305ead47ee01dc7d93e1937cbf50204e8603322138" exitCode=0
Nov 03 01:04:43.688990 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:43.688831    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8" event={"ID":"c04bba05-aee0-497f-b36f-4f91659da630","Type":"ContainerDied","Data":"4142e6f84257b8948a3ff1305ead47ee01dc7d93e1937cbf50204e8603322138"}
Nov 03 01:04:44.890684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.890650    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:44.950977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.950879    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt\" (UniqueName: \"kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt\") pod \"c04bba05-aee0-497f-b36f-4f91659da630\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") "
Nov 03 01:04:44.951167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.951059    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca\") pod \"c04bba05-aee0-497f-b36f-4f91659da630\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") "
Nov 03 01:04:44.951167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.951104    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca\") pod \"c04bba05-aee0-497f-b36f-4f91659da630\" (UID: \"c04bba05-aee0-497f-b36f-4f91659da630\") "
Nov 03 01:04:44.951770 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.951742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca" pod "c04bba05-aee0-497f-b36f-4f91659da630" (UID: "c04bba05-aee0-497f-b36f-4f91659da630"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:04:44.952152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.952127    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:44.953880 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.953855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt" pod "c04bba05-aee0-497f-b36f-4f91659da630" (UID: "c04bba05-aee0-497f-b36f-4f91659da630"). InnerVolumeSpecName "kube-api-access-dpwmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:44.955957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.955919    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca" pod "c04bba05-aee0-497f-b36f-4f91659da630" (UID: "c04bba05-aee0-497f-b36f-4f91659da630"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:44.983281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:44.983238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:45.069463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:45.069164    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwmt\" (UniqueName: \"kubernetes.io/projected/c04bba05-aee0-497f-b36f-4f91659da630-kube-api-access-dpwmt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:45.069463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:45.069211    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04bba05-aee0-497f-b36f-4f91659da630-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:45.709508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:45.709466    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8" event={"ID":"c04bba05-aee0-497f-b36f-4f91659da630","Type":"ContainerDied","Data":"ca280416b2d551e5c3d8e33a344982b0ecf5ae4fd1e3b01bef184ff15c346aa9"}
Nov 03 01:04:45.709508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:45.709493    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"
Nov 03 01:04:45.709740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:45.709537    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca280416b2d551e5c3d8e33a344982b0ecf5ae4fd1e3b01bef184ff15c346aa9"
Nov 03 01:04:46.665039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:46.665005    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:48.340269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:48.340218    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:50.011886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:50.011847    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:51.737140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:51.737097    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:52.437848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:52.437808    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:53.419720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:53.419685    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:53.520593 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:53.520557    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:54.031559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:54.031521    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"]
Nov 03 01:04:54.042842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:54.042804    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9c38501-2xcgm"]
Nov 03 01:04:54.744937 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:04:54.744723    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:04:54.965187 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:54.965146    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:55.073516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:55.068703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:55.384475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:55.384366    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:55.810881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:55.810848    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843a9551-8b8c-420e-aa9c-166fad1f67ce" path="/var/lib/kubelet/pods/843a9551-8b8c-420e-aa9c-166fad1f67ce/volumes"
Nov 03 01:04:55.987183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:55.987143    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz" podUID="e472ade3-5a80-4839-8704-e627869fae14" containerName="main" containerID="cri-o://55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae" gracePeriod=45
Nov 03 01:04:56.289885 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.289851    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 01:04:56.455919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.455872    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache\") pod \"e472ade3-5a80-4839-8704-e627869fae14\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") "
Nov 03 01:04:56.456123 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456004    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs\" (UniqueName: \"kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs\") pod \"e472ade3-5a80-4839-8704-e627869fae14\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") "
Nov 03 01:04:56.456123 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456047    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir\") pod \"e472ade3-5a80-4839-8704-e627869fae14\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") "
Nov 03 01:04:56.456123 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456078    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm\") pod \"e472ade3-5a80-4839-8704-e627869fae14\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") "
Nov 03 01:04:56.456314 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456142    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh\" (UniqueName: \"kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh\") pod \"e472ade3-5a80-4839-8704-e627869fae14\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") "
Nov 03 01:04:56.456314 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456197    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home\") pod \"e472ade3-5a80-4839-8704-e627869fae14\" (UID: \"e472ade3-5a80-4839-8704-e627869fae14\") "
Nov 03 01:04:56.456314 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456193    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache" pod "e472ade3-5a80-4839-8704-e627869fae14" (UID: "e472ade3-5a80-4839-8704-e627869fae14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:56.456534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456326    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir" pod "e472ade3-5a80-4839-8704-e627869fae14" (UID: "e472ade3-5a80-4839-8704-e627869fae14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:56.456658 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home" pod "e472ade3-5a80-4839-8704-e627869fae14" (UID: "e472ade3-5a80-4839-8704-e627869fae14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:56.456949 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456933    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:56.457021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456954    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:56.457021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.456970    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:56.458408 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.458386    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm" pod "e472ade3-5a80-4839-8704-e627869fae14" (UID: "e472ade3-5a80-4839-8704-e627869fae14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:56.458505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.458447    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs" pod "e472ade3-5a80-4839-8704-e627869fae14" (UID: "e472ade3-5a80-4839-8704-e627869fae14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:56.458560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.458545    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh" pod "e472ade3-5a80-4839-8704-e627869fae14" (UID: "e472ade3-5a80-4839-8704-e627869fae14"). InnerVolumeSpecName "kube-api-access-vbhlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:56.557988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.557953    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e472ade3-5a80-4839-8704-e627869fae14-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:56.557988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.557983    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e472ade3-5a80-4839-8704-e627869fae14-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:56.557988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.557994    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhlh\" (UniqueName: \"kubernetes.io/projected/e472ade3-5a80-4839-8704-e627869fae14-kube-api-access-vbhlh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:04:56.726417 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.726326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:56.828655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.828616    4550 generic.go:356] "Generic (PLEG): container finished" podID="e472ade3-5a80-4839-8704-e627869fae14" containerID="55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae" exitCode=0
Nov 03 01:04:56.829210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.828670    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"
Nov 03 01:04:56.829210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.828672    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz" event={"ID":"e472ade3-5a80-4839-8704-e627869fae14","Type":"ContainerDied","Data":"55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae"}
Nov 03 01:04:56.829210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.828723    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz" event={"ID":"e472ade3-5a80-4839-8704-e627869fae14","Type":"ContainerDied","Data":"806c744d65001a81f6b53ec4b10829f4a1291cb3b155ad99d31d6e9128be480d"}
Nov 03 01:04:56.829210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.828745    4550 scope.go:122] "RemoveContainer" containerID="55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae"
Nov 03 01:04:56.842039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.842008    4550 scope.go:122] "RemoveContainer" containerID="55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae"
Nov 03 01:04:56.842322 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:04:56.842292    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae\": container with ID starting with 55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae not found: ID does not exist" containerID="55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae"
Nov 03 01:04:56.842392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.842331    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae"} err="failed to get container status \"55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae\": rpc error: code = NotFound desc = could not find container \"55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae\": container with ID starting with 55e431169c5768a7598e783799e2b77255fa91c73b64982be451299e8a5306ae not found: ID does not exist"
Nov 03 01:04:56.887017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.886980    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"]
Nov 03 01:04:56.900065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:56.900030    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-49d7e5/test-model-49d7e5-kserve-5bd87ddd55-g7hgz"]
Nov 03 01:04:57.809923 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:57.809886    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e472ade3-5a80-4839-8704-e627869fae14" path="/var/lib/kubelet/pods/e472ade3-5a80-4839-8704-e627869fae14/volumes"
Nov 03 01:04:58.397200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:58.397165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:04:59.853255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:04:59.853218    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:00.210670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:00.210629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:01.415557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:01.415504    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:01.515144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:01.514926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:01.742662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:01.742559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:02.218451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:02.218396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:03.996191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:03.996141    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:05.712035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:05.711996    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:07.342286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:07.342250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:07.506673 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:07.506629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:08.909269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:08.909236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:09.686758 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:09.686702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:11.463768 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:11.463735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:13.101860 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:13.101823    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:13.204563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:13.204525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:13.868253 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:13.868133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:14.280004 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:14.279964    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"]
Nov 03 01:05:14.280755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:14.280267    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy" containerID="cri-o://03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b" gracePeriod=30
Nov 03 01:05:14.324085 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:14.324045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:14.854977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:14.854941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:15.094527 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:15.094491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:15.223459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:15.222994    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:17.106092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:17.106059    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:18.716479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:18.716419    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:19.605032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.604999    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 01:05:19.631020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.630985    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631121    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631161    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631236    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631320    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:19.631484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631409    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631573 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631481    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631573 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631521    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631573 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631560    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631580    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:19.631746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631653    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:19.631874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631759    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket\") pod \"e203d55a-e393-443b-b9f3-ac898417f112\" (UID: \"e203d55a-e393-443b-b9f3-ac898417f112\") "
Nov 03 01:05:19.631995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.631974    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:19.632138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.632114    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:19.632798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.632684    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.632798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.632716    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.632798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.632732    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.632798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.632745    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e203d55a-e393-443b-b9f3-ac898417f112-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.632798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.632758    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.633981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.633959    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:19.634530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.634482    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:05:19.634659 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.634631    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "kube-api-access-p4pm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:19.634720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.634673    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token" pod "e203d55a-e393-443b-b9f3-ac898417f112" (UID: "e203d55a-e393-443b-b9f3-ac898417f112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:19.734763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.734723    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pm7\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-kube-api-access-p4pm7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.734763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.734761    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e203d55a-e393-443b-b9f3-ac898417f112-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.735210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.734776    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e203d55a-e393-443b-b9f3-ac898417f112-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:19.735210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:19.734787    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e203d55a-e393-443b-b9f3-ac898417f112-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:05:20.026258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.026224    4550 generic.go:356] "Generic (PLEG): container finished" podID="e203d55a-e393-443b-b9f3-ac898417f112" containerID="03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b" exitCode=0
Nov 03 01:05:20.026493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.026275    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"
Nov 03 01:05:20.026493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.026300    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" event={"ID":"e203d55a-e393-443b-b9f3-ac898417f112","Type":"ContainerDied","Data":"03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b"}
Nov 03 01:05:20.026493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.026364    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs" event={"ID":"e203d55a-e393-443b-b9f3-ac898417f112","Type":"ContainerDied","Data":"7382b5ed34da39c09f6a178158797edb04bdd5c77671274263cf46c36eb93dbd"}
Nov 03 01:05:20.026493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.026390    4550 scope.go:122] "RemoveContainer" containerID="03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b"
Nov 03 01:05:20.037547 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.037524    4550 scope.go:122] "RemoveContainer" containerID="2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"
Nov 03 01:05:20.049195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.049168    4550 scope.go:122] "RemoveContainer" containerID="03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b"
Nov 03 01:05:20.049581 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:05:20.049555    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b\": container with ID starting with 03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b not found: ID does not exist" containerID="03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b"
Nov 03 01:05:20.049693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.049589    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b"} err="failed to get container status \"03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b\": rpc error: code = NotFound desc = could not find container \"03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b\": container with ID starting with 03b05c40f8acd52350c8a787f747283d0cbfa3103cef29ac8a7ca2323727445b not found: ID does not exist"
Nov 03 01:05:20.049693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.049607    4550 scope.go:122] "RemoveContainer" containerID="2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"
Nov 03 01:05:20.049916 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:05:20.049899    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a\": container with ID starting with 2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a not found: ID does not exist" containerID="2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"
Nov 03 01:05:20.049964 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.049920    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a"} err="failed to get container status \"2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a\": rpc error: code = NotFound desc = could not find container \"2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a\": container with ID starting with 2a8538b7590a737968ab3a83fb181ad2f83309ac79947888ebea3ac21bb08c8a not found: ID does not exist"
Nov 03 01:05:20.082749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.082646    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"]
Nov 03 01:05:20.086991 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.086957    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-49d7e5-openshift-default-6567775c79-hh7cs"]
Nov 03 01:05:20.408811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.408776    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:20.533483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.533137    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:20.737598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:20.737501    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:21.540182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:21.540146    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:21.810064 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:21.809983    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e203d55a-e393-443b-b9f3-ac898417f112" path="/var/lib/kubelet/pods/e203d55a-e393-443b-b9f3-ac898417f112/volumes"
Nov 03 01:05:22.170331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:22.170285    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:22.835849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:22.835757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:24.420816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:24.420766    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:24.726549 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:05:24.726463    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:05:26.216274 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:26.216230    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:27.446275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:27.446238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:27.914333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:27.914285    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:29.587979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:29.587856    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:30.742610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:30.742566    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:30.853199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:30.853165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:31.341091 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:31.341051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:31.542186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:31.542143    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:31.950288 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:31.950254    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:32.752643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:32.752606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:33.312491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:33.312458    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:34.989580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:34.989543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:36.634542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:36.634498    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:38.329864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:38.329825    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:38.579706 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:38.579665    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"]
Nov 03 01:05:38.664544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:38.664308    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:38.763663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:38.763631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:39.797735 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:39.797694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:40.134844 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:40.134746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:41.810644 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:41.810604    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:43.542244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:43.542192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:43.580627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:43.580571    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerName="payload-pre-processing" containerID="cri-o://9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e" gracePeriod=25
Nov 03 01:05:45.124785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:45.124739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:45.310139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:45.310100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:46.808292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:46.808250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:48.424526 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:48.424491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:48.996636 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:48.996594    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:05:50.139060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:50.139026    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:51.735561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:51.735522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:53.429153 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:53.429114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:54.725150 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:05:54.725105    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:05:55.017139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:55.017050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:56.744335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:56.744298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:58.432273 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:58.432227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:05:58.996729 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:05:58.996690    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:06:00.115720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:00.115637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:00.847796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:00.847756    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:00.948541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:00.948505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:01.794697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:01.794658    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:03.538901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.538866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:03.887605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.887574    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 01:06:03.992714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.992675    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr\") pod \"69cf8a78-fe36-421d-aed5-a53226509f6e\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") "
Nov 03 01:06:03.992926 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.992815    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume\") pod \"69cf8a78-fe36-421d-aed5-a53226509f6e\" (UID: \"69cf8a78-fe36-421d-aed5-a53226509f6e\") "
Nov 03 01:06:03.993229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.993197    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume" pod "69cf8a78-fe36-421d-aed5-a53226509f6e" (UID: "69cf8a78-fe36-421d-aed5-a53226509f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:03.993514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.993496    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cf8a78-fe36-421d-aed5-a53226509f6e-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:03.994979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:03.994957    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr" pod "69cf8a78-fe36-421d-aed5-a53226509f6e" (UID: "69cf8a78-fe36-421d-aed5-a53226509f6e"). InnerVolumeSpecName "kube-api-access-t8wdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:04.095393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.095348    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wdr\" (UniqueName: \"kubernetes.io/projected/69cf8a78-fe36-421d-aed5-a53226509f6e-kube-api-access-t8wdr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:04.448134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.448098    4550 generic.go:356] "Generic (PLEG): container finished" podID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerID="9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e" exitCode=0
Nov 03 01:06:04.448342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.448155    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"
Nov 03 01:06:04.448342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.448171    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" event={"ID":"69cf8a78-fe36-421d-aed5-a53226509f6e","Type":"ContainerDied","Data":"9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e"}
Nov 03 01:06:04.448342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.448230    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j" event={"ID":"69cf8a78-fe36-421d-aed5-a53226509f6e","Type":"ContainerDied","Data":"fdb33b69469f012c46109d4edd50e0182eeea55f33222aa2006f2932e92979f6"}
Nov 03 01:06:04.448342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.448253    4550 scope.go:122] "RemoveContainer" containerID="9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e"
Nov 03 01:06:04.461326 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.461306    4550 scope.go:122] "RemoveContainer" containerID="9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e"
Nov 03 01:06:04.461683 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:06:04.461657    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e\": container with ID starting with 9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e not found: ID does not exist" containerID="9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e"
Nov 03 01:06:04.461788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.461692    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e"} err="failed to get container status \"9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e\": rpc error: code = NotFound desc = could not find container \"9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e\": container with ID starting with 9adf7855334d29ccdeb1a09b63c3fe2668801e93c06c6f5c57174f905375673e not found: ID does not exist"
Nov 03 01:06:04.503882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.503842    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"]
Nov 03 01:06:04.517063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:04.517029    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-056a96-68fcb4b5c9-h8n6j"]
Nov 03 01:06:05.226811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:05.226775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:05.812230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:05.812193    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" path="/var/lib/kubelet/pods/69cf8a78-fe36-421d-aed5-a53226509f6e/volumes"
Nov 03 01:06:06.906757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:06.906722    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:08.633185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:08.633149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:10.306198 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:10.306160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:11.695145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:11.695115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:12.470643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:12.470606    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:14.131176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:14.131140    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:14.956583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:14.956205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:15.354728 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:15.354618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:15.453127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:15.453090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:15.808266 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:15.808217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:16.133515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:16.133416    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:16.509775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:16.509741    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"]
Nov 03 01:06:16.510612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:16.510581    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy" containerID="cri-o://7686334f0bb0660b40d9bbd046a682abed2b68217001edde795b655607d3cb50" gracePeriod=30
Nov 03 01:06:16.562018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:16.561982    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:17.461206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:17.461171    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:17.595005 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:17.594965    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:17.790683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:17.790587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:19.486827 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:19.486792    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:20.882345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:20.882300    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:21.722463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.722413    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:21.855927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.855542    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 01:06:21.915349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915312    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.915853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915368    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.915853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915524    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.915853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915625    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.915853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915688    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.915853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915758    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.915853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915788    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.916173 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915924    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.916173 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.915950    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj\") pod \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\" (UID: \"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a\") "
Nov 03 01:06:21.917089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.917040    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:21.917304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.917233    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:21.917304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.917260    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:21.917554 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.917416    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:21.918178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.918156    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.918318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.918183    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.918318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.918198    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.918318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.918211    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.918559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.918471    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:21.931805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.931750    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:21.932181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.932153    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "kube-api-access-jd2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:21.932393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.932346    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:21.933198 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:21.933170    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo" pod "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" (UID: "158a96eb-7d12-4eec-bfd8-e91c95a1ef4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:22.020100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.020047    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:22.020100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.020096    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2bj\" (UniqueName: \"kubernetes.io/projected/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-kube-api-access-jd2bj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:22.020100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.020107    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:22.020100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.020116    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:22.020406 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.020124    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:22.461523 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.461488    4550 scope.go:122] "RemoveContainer" containerID="6761bec57d1b47678c389457d3985f2dbc8a8aa10ea0865dab68aaf0bd194d97"
Nov 03 01:06:22.473612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.473591    4550 scope.go:122] "RemoveContainer" containerID="7686334f0bb0660b40d9bbd046a682abed2b68217001edde795b655607d3cb50"
Nov 03 01:06:22.484188 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.484161    4550 scope.go:122] "RemoveContainer" containerID="1202e1c4e8384f56a2b0989a39c65ce160528ee2e3acc28d7663d1f086b9a237"
Nov 03 01:06:22.615391 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.615342    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" event={"ID":"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a","Type":"ContainerDied","Data":"7686334f0bb0660b40d9bbd046a682abed2b68217001edde795b655607d3cb50"}
Nov 03 01:06:22.615616 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.615424    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k" event={"ID":"158a96eb-7d12-4eec-bfd8-e91c95a1ef4a","Type":"ContainerDied","Data":"2ed8afbdc83f5160322ffc3d34cf1038bf39b4d82af637a4b97ae49f5104b9c7"}
Nov 03 01:06:22.615616 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.615476    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"
Nov 03 01:06:22.716492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.716397    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"]
Nov 03 01:06:22.720297 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:22.720272    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7f20f7-openshift-default-87bd658df-v977k"]
Nov 03 01:06:23.316895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:23.316860    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:23.811341 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:23.811301    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" path="/var/lib/kubelet/pods/158a96eb-7d12-4eec-bfd8-e91c95a1ef4a/volumes"
Nov 03 01:06:24.042839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:24.042798    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:24.148183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:24.148077    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:24.726408 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:06:24.726364    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:06:25.221286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:25.221248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:25.339457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:25.338185    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:26.767871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:26.767835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:26.909655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:26.909617    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:27.009942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.009902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:27.143977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.143894    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"]
Nov 03 01:06:27.144617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144597    4550 cpu_manager.go:414] "containerMap: removing container" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerName="payload-pre-processing"
Nov 03 01:06:27.144617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144615    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerName="payload-pre-processing"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144625    4550 cpu_manager.go:414] "containerMap: removing container" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144630    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144647    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144652    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144668    4550 cpu_manager.go:414] "containerMap: removing container" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144673    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144693    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144698    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144715    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e472ade3-5a80-4839-8704-e627869fae14" containerName="main"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144720    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e472ade3-5a80-4839-8704-e627869fae14" containerName="main"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144730    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c04bba05-aee0-497f-b36f-4f91659da630" containerName="revoke-keys"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144735    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04bba05-aee0-497f-b36f-4f91659da630" containerName="revoke-keys"
Nov 03 01:06:27.144824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144830    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e472ade3-5a80-4839-8704-e627869fae14" containerName="main"
Nov 03 01:06:27.145618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144840    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04bba05-aee0-497f-b36f-4f91659da630" containerName="revoke-keys"
Nov 03 01:06:27.145618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144850    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="69cf8a78-fe36-421d-aed5-a53226509f6e" containerName="payload-pre-processing"
Nov 03 01:06:27.145618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144857    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy"
Nov 03 01:06:27.145618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144866    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy"
Nov 03 01:06:27.145618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.144873    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="158a96eb-7d12-4eec-bfd8-e91c95a1ef4a" containerName="istio-proxy"
Nov 03 01:06:27.148790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.148770    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.168044 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.167721    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"]
Nov 03 01:06:27.172922 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.172891    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"]
Nov 03 01:06:27.173814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.173793    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e203d55a-e393-443b-b9f3-ac898417f112" containerName="istio-proxy"
Nov 03 01:06:27.175242 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.175218    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:06:27.175401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.175304    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:06:27.178225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.178199    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.195671 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.195644    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"]
Nov 03 01:06:27.205939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.205907    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.206097 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.206043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.206221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.206203    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlbnd\" (UniqueName: \"kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.308098 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.308056    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.308305 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.308134    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.308305 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.308194    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.308381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.308356    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlbnd\" (UniqueName: \"kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.308507 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.308493    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494cr\" (UniqueName: \"kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.308566 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.308548    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.309167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.309147    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.310619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.310595    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.329192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.329165    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlbnd\" (UniqueName: \"kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd\") pod \"maas-api-revoke-sub-715484be-5tk98\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.409804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.409767    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.409992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.409873    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.410089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.410062    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494cr\" (UniqueName: \"kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.410552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.410522    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.412464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.412407    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.431958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.431930    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-494cr\" (UniqueName: \"kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr\") pod \"maas-api-revoke-sub-19aafc99-kk4dj\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.473427 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.473393    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:27.498998 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.498966    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:27.499800 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.499774    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:27.789860 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:06:27.789830    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894553c8_7dde_46d1_90f5_b68d5c772127.slice/crio-d7e08ac048ede59fdd4ac8d1fbc35862b592d37013351e38e951daaae4b88a2a WatchSource:0}: Error finding container d7e08ac048ede59fdd4ac8d1fbc35862b592d37013351e38e951daaae4b88a2a: Status 404 returned error can't find the container with id d7e08ac048ede59fdd4ac8d1fbc35862b592d37013351e38e951daaae4b88a2a
Nov 03 01:06:27.792261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.792234    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"]
Nov 03 01:06:27.844195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.844154    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"]
Nov 03 01:06:27.844485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.844456    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy" containerID="cri-o://e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce" gracePeriod=30
Nov 03 01:06:27.871095 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.871059    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:27.874493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:27.874401    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"]
Nov 03 01:06:28.277903 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.277870    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:28.676695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.676652    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98" event={"ID":"894553c8-7dde-46d1-90f5-b68d5c772127","Type":"ContainerStarted","Data":"29f849d9b33a5ce1095fa2492af2d6425cfbc9dc2a6511d2bfb4978207269939"}
Nov 03 01:06:28.677328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.676725    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98" event={"ID":"894553c8-7dde-46d1-90f5-b68d5c772127","Type":"ContainerStarted","Data":"d7e08ac048ede59fdd4ac8d1fbc35862b592d37013351e38e951daaae4b88a2a"}
Nov 03 01:06:28.678480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.678393    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj" event={"ID":"735f6cd4-6968-4f31-a1be-a1d704473a0d","Type":"ContainerStarted","Data":"43ce1673ff2129c43a3052d70aa593c39780d22383110c88542341bcb26be8f9"}
Nov 03 01:06:28.678480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.678471    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj" event={"ID":"735f6cd4-6968-4f31-a1be-a1d704473a0d","Type":"ContainerStarted","Data":"6b6a0c490a99799e13f11963c8bdb6974366b561b6320f17fdd6a87f1a55fed2"}
Nov 03 01:06:28.721800 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.721742    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98" podStartSLOduration=1.72172933 podStartE2EDuration="1.72172933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:28.720391108 +0000 UTC m=+3671.041927925" watchObservedRunningTime="2026-11-03 01:06:28.72172933 +0000 UTC m=+3671.043266128"
Nov 03 01:06:28.777104 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.777049    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj" podStartSLOduration=1.7770344219999998 podStartE2EDuration="1.777034422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:28.772189779 +0000 UTC m=+3671.093726598" watchObservedRunningTime="2026-11-03 01:06:28.777034422 +0000 UTC m=+3671.098571220"
Nov 03 01:06:28.795423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:28.795385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:29.011780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.011685    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:29.170028 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.169996    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"]
Nov 03 01:06:29.174536 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.174517    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.195131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.195104    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"]
Nov 03 01:06:29.223479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.223305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:29.264314 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.264231    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbc4m\" (UniqueName: \"kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.264534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.264394    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.264647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.264628    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.366409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.366353    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.366625 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.366516    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.366773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.366754    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbc4m\" (UniqueName: \"kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.367381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.367348    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.369016 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.368990    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.386970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.386942    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbc4m\" (UniqueName: \"kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m\") pod \"maas-api-revoke-sub-b7a66913-dv5fg\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.489243 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.489208    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:29.690858 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.690818    4550 generic.go:356] "Generic (PLEG): container finished" podID="894553c8-7dde-46d1-90f5-b68d5c772127" containerID="29f849d9b33a5ce1095fa2492af2d6425cfbc9dc2a6511d2bfb4978207269939" exitCode=0
Nov 03 01:06:29.691193 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.690925    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98" event={"ID":"894553c8-7dde-46d1-90f5-b68d5c772127","Type":"ContainerDied","Data":"29f849d9b33a5ce1095fa2492af2d6425cfbc9dc2a6511d2bfb4978207269939"}
Nov 03 01:06:29.693407 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.693377    4550 generic.go:356] "Generic (PLEG): container finished" podID="735f6cd4-6968-4f31-a1be-a1d704473a0d" containerID="43ce1673ff2129c43a3052d70aa593c39780d22383110c88542341bcb26be8f9" exitCode=0
Nov 03 01:06:29.693588 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.693484    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj" event={"ID":"735f6cd4-6968-4f31-a1be-a1d704473a0d","Type":"ContainerDied","Data":"43ce1673ff2129c43a3052d70aa593c39780d22383110c88542341bcb26be8f9"}
Nov 03 01:06:29.813538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:29.813513    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"]
Nov 03 01:06:30.706054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:30.706013    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg" event={"ID":"7fc62ca5-e313-477d-8496-33a5fed12477","Type":"ContainerStarted","Data":"f6d4ec65d3f3d4575619ef1e5b1ebc3879fc0abd03addbaf4c19696fd18d7e59"}
Nov 03 01:06:30.706262 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:30.706080    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg" event={"ID":"7fc62ca5-e313-477d-8496-33a5fed12477","Type":"ContainerStarted","Data":"dd575fa64907dcd4e743c74101bb076f5a4920afcc5e77acfb3bf4eaf0909420"}
Nov 03 01:06:30.738840 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:30.738803    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:30.747864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:30.747811    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg" podStartSLOduration=1.7477964689999999 podStartE2EDuration="1.747796469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:30.745010235 +0000 UTC m=+3673.066547045" watchObservedRunningTime="2026-11-03 01:06:30.747796469 +0000 UTC m=+3673.069333282"
Nov 03 01:06:30.948405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:30.948378    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:30.972135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:30.972107    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:31.105180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105144    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca\") pod \"735f6cd4-6968-4f31-a1be-a1d704473a0d\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") "
Nov 03 01:06:31.105180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105187    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd\" (UniqueName: \"kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd\") pod \"894553c8-7dde-46d1-90f5-b68d5c772127\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") "
Nov 03 01:06:31.105483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105267    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca\") pod \"894553c8-7dde-46d1-90f5-b68d5c772127\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") "
Nov 03 01:06:31.105483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105359    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca\") pod \"894553c8-7dde-46d1-90f5-b68d5c772127\" (UID: \"894553c8-7dde-46d1-90f5-b68d5c772127\") "
Nov 03 01:06:31.105483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105470    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr\" (UniqueName: \"kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr\") pod \"735f6cd4-6968-4f31-a1be-a1d704473a0d\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") "
Nov 03 01:06:31.105670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105595    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca" pod "735f6cd4-6968-4f31-a1be-a1d704473a0d" (UID: "735f6cd4-6968-4f31-a1be-a1d704473a0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:31.105670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105627    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca\") pod \"735f6cd4-6968-4f31-a1be-a1d704473a0d\" (UID: \"735f6cd4-6968-4f31-a1be-a1d704473a0d\") "
Nov 03 01:06:31.105824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.105803    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca" pod "894553c8-7dde-46d1-90f5-b68d5c772127" (UID: "894553c8-7dde-46d1-90f5-b68d5c772127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:31.106362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.106338    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:31.106495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.106385    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:31.107680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.107652    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd" pod "894553c8-7dde-46d1-90f5-b68d5c772127" (UID: "894553c8-7dde-46d1-90f5-b68d5c772127"). InnerVolumeSpecName "kube-api-access-nlbnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:31.107811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.107748    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr" pod "735f6cd4-6968-4f31-a1be-a1d704473a0d" (UID: "735f6cd4-6968-4f31-a1be-a1d704473a0d"). InnerVolumeSpecName "kube-api-access-494cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:31.108117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.108095    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca" pod "735f6cd4-6968-4f31-a1be-a1d704473a0d" (UID: "735f6cd4-6968-4f31-a1be-a1d704473a0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:31.108174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.108096    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca" pod "894553c8-7dde-46d1-90f5-b68d5c772127" (UID: "894553c8-7dde-46d1-90f5-b68d5c772127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:31.207663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.207630    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-494cr\" (UniqueName: \"kubernetes.io/projected/735f6cd4-6968-4f31-a1be-a1d704473a0d-kube-api-access-494cr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:31.207663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.207661    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/735f6cd4-6968-4f31-a1be-a1d704473a0d-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:31.207663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.207671    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlbnd\" (UniqueName: \"kubernetes.io/projected/894553c8-7dde-46d1-90f5-b68d5c772127-kube-api-access-nlbnd\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:31.207932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.207680    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/894553c8-7dde-46d1-90f5-b68d5c772127-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:31.714835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.714799    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98" event={"ID":"894553c8-7dde-46d1-90f5-b68d5c772127","Type":"ContainerDied","Data":"d7e08ac048ede59fdd4ac8d1fbc35862b592d37013351e38e951daaae4b88a2a"}
Nov 03 01:06:31.715034 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.714860    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7e08ac048ede59fdd4ac8d1fbc35862b592d37013351e38e951daaae4b88a2a"
Nov 03 01:06:31.715034 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.714803    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"
Nov 03 01:06:31.716475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.716454    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"
Nov 03 01:06:31.716579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.716493    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj" event={"ID":"735f6cd4-6968-4f31-a1be-a1d704473a0d","Type":"ContainerDied","Data":"6b6a0c490a99799e13f11963c8bdb6974366b561b6320f17fdd6a87f1a55fed2"}
Nov 03 01:06:31.716579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.716540    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b6a0c490a99799e13f11963c8bdb6974366b561b6320f17fdd6a87f1a55fed2"
Nov 03 01:06:31.718315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.718292    4550 generic.go:356] "Generic (PLEG): container finished" podID="7fc62ca5-e313-477d-8496-33a5fed12477" containerID="f6d4ec65d3f3d4575619ef1e5b1ebc3879fc0abd03addbaf4c19696fd18d7e59" exitCode=0
Nov 03 01:06:31.718466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:31.718379    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg" event={"ID":"7fc62ca5-e313-477d-8496-33a5fed12477","Type":"ContainerDied","Data":"f6d4ec65d3f3d4575619ef1e5b1ebc3879fc0abd03addbaf4c19696fd18d7e59"}
Nov 03 01:06:32.305920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:32.305887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:32.984697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:32.984618    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:33.032814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.032720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca\") pod \"7fc62ca5-e313-477d-8496-33a5fed12477\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") "
Nov 03 01:06:33.033018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.032982    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca\") pod \"7fc62ca5-e313-477d-8496-33a5fed12477\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") "
Nov 03 01:06:33.036585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.033091    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m\" (UniqueName: \"kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m\") pod \"7fc62ca5-e313-477d-8496-33a5fed12477\" (UID: \"7fc62ca5-e313-477d-8496-33a5fed12477\") "
Nov 03 01:06:33.036585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.034508    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca" pod "7fc62ca5-e313-477d-8496-33a5fed12477" (UID: "7fc62ca5-e313-477d-8496-33a5fed12477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:33.036585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.036135    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca" pod "7fc62ca5-e313-477d-8496-33a5fed12477" (UID: "7fc62ca5-e313-477d-8496-33a5fed12477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:33.048556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.039895    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m" pod "7fc62ca5-e313-477d-8496-33a5fed12477" (UID: "7fc62ca5-e313-477d-8496-33a5fed12477"). InnerVolumeSpecName "kube-api-access-vbc4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:33.135036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.135006    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbc4m\" (UniqueName: \"kubernetes.io/projected/7fc62ca5-e313-477d-8496-33a5fed12477-kube-api-access-vbc4m\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.135036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.135038    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.135232 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.135053    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc62ca5-e313-477d-8496-33a5fed12477-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.137947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.137928    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 01:06:33.235940 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.235847    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.235940 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.235935    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.235960    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236020    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236122    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236199    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.236221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236214    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.236535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236248    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236360    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236394    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236443    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs\") pod \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\" (UID: \"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0\") "
Nov 03 01:06:33.236762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.236616    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:33.237232 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237179    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.237362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237246    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.237635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237611    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.237737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237641    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.237737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237657    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.237737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237670    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.237737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.237685    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.239125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.239094    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:33.239249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.239212    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "kube-api-access-f2dt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:33.239537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.239503    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:33.239693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.239674    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy" pod "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" (UID: "d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.339009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.338970    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.339009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.339005    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2dt5\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-kube-api-access-f2dt5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.339476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.339019    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.339476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.339035    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.734538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.734496    4550 generic.go:356] "Generic (PLEG): container finished" podID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerID="e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce" exitCode=0
Nov 03 01:06:33.734738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.734557    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"
Nov 03 01:06:33.734738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.734581    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" event={"ID":"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0","Type":"ContainerDied","Data":"e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce"}
Nov 03 01:06:33.734738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.734661    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg" event={"ID":"d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0","Type":"ContainerDied","Data":"f7fb2eb372ac08c9c8d7c34c07283ccf0bcb180bdda63080d89e3486abdd75fb"}
Nov 03 01:06:33.734738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.734696    4550 scope.go:122] "RemoveContainer" containerID="e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce"
Nov 03 01:06:33.736406 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.736384    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"
Nov 03 01:06:33.736406 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.736397    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg" event={"ID":"7fc62ca5-e313-477d-8496-33a5fed12477","Type":"ContainerDied","Data":"dd575fa64907dcd4e743c74101bb076f5a4920afcc5e77acfb3bf4eaf0909420"}
Nov 03 01:06:33.736633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.736478    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd575fa64907dcd4e743c74101bb076f5a4920afcc5e77acfb3bf4eaf0909420"
Nov 03 01:06:33.747290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.747263    4550 scope.go:122] "RemoveContainer" containerID="0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"
Nov 03 01:06:33.758321 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.758295    4550 scope.go:122] "RemoveContainer" containerID="e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce"
Nov 03 01:06:33.758823 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:06:33.758797    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce\": container with ID starting with e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce not found: ID does not exist" containerID="e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce"
Nov 03 01:06:33.758902 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.758840    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce"} err="failed to get container status \"e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce\": rpc error: code = NotFound desc = could not find container \"e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce\": container with ID starting with e1a1a721f850c54356573180622da8a495df1aee49b3178fe94ed91abdab02ce not found: ID does not exist"
Nov 03 01:06:33.758902 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.758863    4550 scope.go:122] "RemoveContainer" containerID="0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"
Nov 03 01:06:33.759140 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:06:33.759111    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a\": container with ID starting with 0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a not found: ID does not exist" containerID="0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"
Nov 03 01:06:33.759206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.759143    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a"} err="failed to get container status \"0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a\": rpc error: code = NotFound desc = could not find container \"0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a\": container with ID starting with 0a505a86f6573e35deb34003ac5d4fcd38028457d39f6ad6b0ce1ea04e43cf0a not found: ID does not exist"
Nov 03 01:06:33.812231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.812189    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"]
Nov 03 01:06:33.823094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.823058    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-dfvjg"]
Nov 03 01:06:33.972303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:33.972269    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:35.684138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:35.684091    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:35.792476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:35.792420    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:35.817746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:35.817713    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" path="/var/lib/kubelet/pods/d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0/volumes"
Nov 03 01:06:37.309092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:37.309048    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:37.544440 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:37.544397    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:38.151123 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.151080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:38.638117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638032    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"]
Nov 03 01:06:38.638765 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638747    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638766    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638785    4550 cpu_manager.go:414] "containerMap: removing container" podUID="894553c8-7dde-46d1-90f5-b68d5c772127" containerName="revoke-keys"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638790    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894553c8-7dde-46d1-90f5-b68d5c772127" containerName="revoke-keys"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638805    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7fc62ca5-e313-477d-8496-33a5fed12477" containerName="revoke-keys"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638810    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc62ca5-e313-477d-8496-33a5fed12477" containerName="revoke-keys"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638821    4550 cpu_manager.go:414] "containerMap: removing container" podUID="735f6cd4-6968-4f31-a1be-a1d704473a0d" containerName="revoke-keys"
Nov 03 01:06:38.638831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638825    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735f6cd4-6968-4f31-a1be-a1d704473a0d" containerName="revoke-keys"
Nov 03 01:06:38.639060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638955    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy"
Nov 03 01:06:38.639060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638968    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="894553c8-7dde-46d1-90f5-b68d5c772127" containerName="revoke-keys"
Nov 03 01:06:38.639060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638975    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc62ca5-e313-477d-8496-33a5fed12477" containerName="revoke-keys"
Nov 03 01:06:38.639060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638981    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="735f6cd4-6968-4f31-a1be-a1d704473a0d" containerName="revoke-keys"
Nov 03 01:06:38.639060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.638992    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy"
Nov 03 01:06:38.644483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.644425    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.660831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.660806    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:06:38.662750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.662723    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"]
Nov 03 01:06:38.670495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.669390    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:06:38.745903 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.745862    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.746143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.746031    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.746261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.746232    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mw5z\" (UniqueName: \"kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.847790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.847750    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mw5z\" (UniqueName: \"kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.848002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.847940    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.848080 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.848050    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.848791 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.848768    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.850529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.850509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.868227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.868199    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mw5z\" (UniqueName: \"kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z\") pod \"maas-api-revoke-sub-cc7cc115-4swsd\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:38.972590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:38.972539    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:39.098852 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:39.098815    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:39.282377 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:06:39.282342    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac581c88_6645_4abd_bd78_efcd30d01d9b.slice/crio-df4982e2c1aa0be450fefe58e6ae846d013a3a80bb66bd4cf764bb12be5498c2 WatchSource:0}: Error finding container df4982e2c1aa0be450fefe58e6ae846d013a3a80bb66bd4cf764bb12be5498c2: Status 404 returned error can't find the container with id df4982e2c1aa0be450fefe58e6ae846d013a3a80bb66bd4cf764bb12be5498c2
Nov 03 01:06:39.283116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:39.283091    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"]
Nov 03 01:06:39.789809 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:39.789762    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd" event={"ID":"ac581c88-6645-4abd-bd78-efcd30d01d9b","Type":"ContainerStarted","Data":"94474e31501912db7151a7e8cc1de32c138b00fd156f84d2d8b58bf9fde959d8"}
Nov 03 01:06:39.790330 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:39.789831    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd" event={"ID":"ac581c88-6645-4abd-bd78-efcd30d01d9b","Type":"ContainerStarted","Data":"df4982e2c1aa0be450fefe58e6ae846d013a3a80bb66bd4cf764bb12be5498c2"}
Nov 03 01:06:39.834707 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:39.834649    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd" podStartSLOduration=1.834634854 podStartE2EDuration="1.834634854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:39.833219916 +0000 UTC m=+3682.154756744" watchObservedRunningTime="2026-11-03 01:06:39.834634854 +0000 UTC m=+3682.156171653"
Nov 03 01:06:40.800970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:40.800930    4550 generic.go:356] "Generic (PLEG): container finished" podID="ac581c88-6645-4abd-bd78-efcd30d01d9b" containerID="94474e31501912db7151a7e8cc1de32c138b00fd156f84d2d8b58bf9fde959d8" exitCode=0
Nov 03 01:06:40.801530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:40.800991    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd" event={"ID":"ac581c88-6645-4abd-bd78-efcd30d01d9b","Type":"ContainerDied","Data":"94474e31501912db7151a7e8cc1de32c138b00fd156f84d2d8b58bf9fde959d8"}
Nov 03 01:06:40.950897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:40.950863    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:41.128874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:41.128787    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"]
Nov 03 01:06:41.129048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:41.128991    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy" containerID="cri-o://afaecac6aa515470f844e3c074b28d1c76598f8494427e2eca69f125620d32e4" gracePeriod=30
Nov 03 01:06:41.986462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:41.986420    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:42.105329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.105237    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z\" (UniqueName: \"kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z\") pod \"ac581c88-6645-4abd-bd78-efcd30d01d9b\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") "
Nov 03 01:06:42.105329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.105316    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca\") pod \"ac581c88-6645-4abd-bd78-efcd30d01d9b\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") "
Nov 03 01:06:42.105611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.105496    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca\") pod \"ac581c88-6645-4abd-bd78-efcd30d01d9b\" (UID: \"ac581c88-6645-4abd-bd78-efcd30d01d9b\") "
Nov 03 01:06:42.105768 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.105742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca" pod "ac581c88-6645-4abd-bd78-efcd30d01d9b" (UID: "ac581c88-6645-4abd-bd78-efcd30d01d9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:42.106210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.106185    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:42.107530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.107486    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z" pod "ac581c88-6645-4abd-bd78-efcd30d01d9b" (UID: "ac581c88-6645-4abd-bd78-efcd30d01d9b"). InnerVolumeSpecName "kube-api-access-9mw5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:42.107643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.107584    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca" pod "ac581c88-6645-4abd-bd78-efcd30d01d9b" (UID: "ac581c88-6645-4abd-bd78-efcd30d01d9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:42.207757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.207716    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac581c88-6645-4abd-bd78-efcd30d01d9b-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:42.207757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.207756    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mw5z\" (UniqueName: \"kubernetes.io/projected/ac581c88-6645-4abd-bd78-efcd30d01d9b-kube-api-access-9mw5z\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:42.819738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.819703    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd" event={"ID":"ac581c88-6645-4abd-bd78-efcd30d01d9b","Type":"ContainerDied","Data":"df4982e2c1aa0be450fefe58e6ae846d013a3a80bb66bd4cf764bb12be5498c2"}
Nov 03 01:06:42.819924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.819743    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"
Nov 03 01:06:42.819924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.819755    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df4982e2c1aa0be450fefe58e6ae846d013a3a80bb66bd4cf764bb12be5498c2"
Nov 03 01:06:42.828981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:42.828948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:43.188646 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.188614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:43.702666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.702626    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"]
Nov 03 01:06:43.703749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.703720    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ac581c88-6645-4abd-bd78-efcd30d01d9b" containerName="revoke-keys"
Nov 03 01:06:43.703749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.703746    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac581c88-6645-4abd-bd78-efcd30d01d9b" containerName="revoke-keys"
Nov 03 01:06:43.703984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.703773    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy"
Nov 03 01:06:43.703984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.703781    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6912fc0-c1e0-4bf3-a722-daa8a11d0fd0" containerName="istio-proxy"
Nov 03 01:06:43.703984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.703916    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac581c88-6645-4abd-bd78-efcd30d01d9b" containerName="revoke-keys"
Nov 03 01:06:43.707884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.707863    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.722002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.721974    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"]
Nov 03 01:06:43.729756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.729709    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:06:43.729756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.729734    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:06:43.834841 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.834805    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.835030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.834884    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.835030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.835019    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4rb\" (UniqueName: \"kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.936384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.936343    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.936384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.936398    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.936703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.936526    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4rb\" (UniqueName: \"kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.937154 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.937132    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.938913 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.938891    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.956691 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.956617    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4rb\" (UniqueName: \"kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb\") pod \"maas-api-revoke-sub-869af6fd-qtltp\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:43.983503 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:43.983454    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:44.023596 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:44.023560    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:44.333388 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:06:44.333344    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e08195_7ad1_4b3f_9a8d_d57288aea242.slice/crio-ca7c7b60cc68f00807409952fd34a3328dbbd77642d6a944e9eebd3b23ca9bc9 WatchSource:0}: Error finding container ca7c7b60cc68f00807409952fd34a3328dbbd77642d6a944e9eebd3b23ca9bc9: Status 404 returned error can't find the container with id ca7c7b60cc68f00807409952fd34a3328dbbd77642d6a944e9eebd3b23ca9bc9
Nov 03 01:06:44.335215 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:44.335188    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"]
Nov 03 01:06:44.589817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:44.589740    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:44.837219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:44.837180    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp" event={"ID":"33e08195-7ad1-4b3f-9a8d-d57288aea242","Type":"ContainerStarted","Data":"4b28c045b75826c31a6eb40f948d7c917e4ac76b22118c940340264ebbab4453"}
Nov 03 01:06:44.837466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:44.837246    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp" event={"ID":"33e08195-7ad1-4b3f-9a8d-d57288aea242","Type":"ContainerStarted","Data":"ca7c7b60cc68f00807409952fd34a3328dbbd77642d6a944e9eebd3b23ca9bc9"}
Nov 03 01:06:44.877083 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:44.876977    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp" podStartSLOduration=1.876960773 podStartE2EDuration="1.876960773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:44.876035521 +0000 UTC m=+3687.197572320" watchObservedRunningTime="2026-11-03 01:06:44.876960773 +0000 UTC m=+3687.198497571"
Nov 03 01:06:45.106869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:45.106830    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:45.848333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:45.848294    4550 generic.go:356] "Generic (PLEG): container finished" podID="33e08195-7ad1-4b3f-9a8d-d57288aea242" containerID="4b28c045b75826c31a6eb40f948d7c917e4ac76b22118c940340264ebbab4453" exitCode=0
Nov 03 01:06:45.848779 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:45.848373    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp" event={"ID":"33e08195-7ad1-4b3f-9a8d-d57288aea242","Type":"ContainerDied","Data":"4b28c045b75826c31a6eb40f948d7c917e4ac76b22118c940340264ebbab4453"}
Nov 03 01:06:46.324412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:46.324362    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:46.909359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:46.898952    4550 generic.go:356] "Generic (PLEG): container finished" podID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerID="afaecac6aa515470f844e3c074b28d1c76598f8494427e2eca69f125620d32e4" exitCode=0
Nov 03 01:06:46.909359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:46.899393    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" event={"ID":"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac","Type":"ContainerDied","Data":"afaecac6aa515470f844e3c074b28d1c76598f8494427e2eca69f125620d32e4"}
Nov 03 01:06:47.023458 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.017972    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045392    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045533    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045629    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045730    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045766    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045812    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045845    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045952    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.049463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.045991    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x\") pod \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\" (UID: \"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac\") "
Nov 03 01:06:47.073456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.065784    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.073456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.066186    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.073456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.071667    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:47.073456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.071750    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.096512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.079239    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.130739 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.129754    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:47.183015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.157118    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.183015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.157155    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.183015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.157170    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.183015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.157183    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.183015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.157196    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.183015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.157209    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.254513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.233899    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:47.254513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.235064    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "kube-api-access-5sd4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:47.274463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.258811    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sd4x\" (UniqueName: \"kubernetes.io/projected/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-kube-api-access-5sd4x\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.274463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.258852    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:47.281241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.281125    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo" pod "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" (UID: "dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:47.403478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:47.361818    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.405344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.401541    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq" event={"ID":"dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac","Type":"ContainerDied","Data":"927ab1cae1a7f55579c927fa5f0f7e9ffb7d73d807b22641d3108dda8507a3dd"}
Nov 03 01:06:48.405344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.401621    4550 scope.go:122] "RemoveContainer" containerID="afaecac6aa515470f844e3c074b28d1c76598f8494427e2eca69f125620d32e4"
Nov 03 01:06:48.405344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.401823    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"
Nov 03 01:06:48.488975 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.486256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Nov 03 01:06:48.527740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.526579    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.578890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.578787    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:48.601462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.588523    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.635425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.631646    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.637762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.637697    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:48.664473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.661258    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.696541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.684766    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.700571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.700398    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca\") pod \"33e08195-7ad1-4b3f-9a8d-d57288aea242\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") "
Nov 03 01:06:48.702147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.701083    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca\") pod \"33e08195-7ad1-4b3f-9a8d-d57288aea242\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") "
Nov 03 01:06:48.702147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.701198    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb\" (UniqueName: \"kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb\") pod \"33e08195-7ad1-4b3f-9a8d-d57288aea242\" (UID: \"33e08195-7ad1-4b3f-9a8d-d57288aea242\") "
Nov 03 01:06:48.717294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.710846    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca" pod "33e08195-7ad1-4b3f-9a8d-d57288aea242" (UID: "33e08195-7ad1-4b3f-9a8d-d57288aea242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:48.717294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.711195    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.870772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.862643    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"]
Nov 03 01:06:48.872074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.871805    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.945024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.930562    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Nov 03 01:06:48.949149 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:48.946463    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:49.016548 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.016502    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-h7zcq"]
Nov 03 01:06:49.068806 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:06:49.068757    4550 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd9175e_57ee_4fe2_9cfb_1e3b2eb4faac.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd9175e_57ee_4fe2_9cfb_1e3b2eb4faac.slice/crio-927ab1cae1a7f55579c927fa5f0f7e9ffb7d73d807b22641d3108dda8507a3dd\": RecentStats: unable to find data in memory cache]"
Nov 03 01:06:49.176129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.116069    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb" pod "33e08195-7ad1-4b3f-9a8d-d57288aea242" (UID: "33e08195-7ad1-4b3f-9a8d-d57288aea242"). InnerVolumeSpecName "kube-api-access-gw4rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:49.176129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.133631    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca" pod "33e08195-7ad1-4b3f-9a8d-d57288aea242" (UID: "33e08195-7ad1-4b3f-9a8d-d57288aea242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:49.218796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.218720    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4rb\" (UniqueName: \"kubernetes.io/projected/33e08195-7ad1-4b3f-9a8d-d57288aea242-kube-api-access-gw4rb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:49.218796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.218762    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e08195-7ad1-4b3f-9a8d-d57288aea242-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:06:49.428831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.428718    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp" event={"ID":"33e08195-7ad1-4b3f-9a8d-d57288aea242","Type":"ContainerDied","Data":"ca7c7b60cc68f00807409952fd34a3328dbbd77642d6a944e9eebd3b23ca9bc9"}
Nov 03 01:06:49.429522 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.429471    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca7c7b60cc68f00807409952fd34a3328dbbd77642d6a944e9eebd3b23ca9bc9"
Nov 03 01:06:49.429746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.429730    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"
Nov 03 01:06:49.810763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:49.810665    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" path="/var/lib/kubelet/pods/dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac/volumes"
Nov 03 01:06:50.426144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:50.426097    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:50.985790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:50.985757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:52.115965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:52.115906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:53.813279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:53.813248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:54.727261 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:06:54.727227    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:06:55.504880 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:55.504839    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:57.439783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:57.439746    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:06:59.102987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:06:59.102946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:00.796225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:00.796183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:02.425773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:02.425726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:03.588017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:03.587976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:03.998617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:03.998582    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:04.110098 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:04.110051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:04.198628 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:04.198591    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:04.435993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:04.435960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:05.850468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:05.850418    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:07.530610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:07.530564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:09.199088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:09.199046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:09.640108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:09.639987    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"]
Nov 03 01:07:09.739650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:09.739609    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:09.874423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:09.860989    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:10.167923 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:10.167883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:11.009061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:11.009023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:11.902170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:11.902134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:12.592348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:12.592314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:12.685601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:12.685563    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:12.945587 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:12.945554    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:14.351304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:14.351265    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:14.642822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:14.642518    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerName="payload-pre-processing" containerID="cri-o://2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84" gracePeriod=25
Nov 03 01:07:14.747416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:14.747375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:15.363452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:15.336377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:15.426751 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:15.426691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:15.565887 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:15.565852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:15.912888 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:15.912849    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:16.260006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:16.259913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:16.371418 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:16.371380    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:17.071075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:17.071028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:17.365691 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:17.365582    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:17.490167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:17.490132    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:17.495737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:17.495700    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"]
Nov 03 01:07:17.496246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:17.496219    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy" containerID="cri-o://3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2" gracePeriod=30
Nov 03 01:07:18.334960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:18.334913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:18.438390 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:18.438359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:19.197876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:19.197835    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:19.415586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:19.415544    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:19.609094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:19.608991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:20.394734 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:20.394698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:21.333550 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:21.333513    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:22.823552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.823523    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 01:07:22.906611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.906567    4550 generic.go:356] "Generic (PLEG): container finished" podID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerID="3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2" exitCode=0
Nov 03 01:07:22.906896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.906870    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" event={"ID":"ee7d5e68-ac10-426f-8a0a-f5b5d791f875","Type":"ContainerDied","Data":"3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2"}
Nov 03 01:07:22.906996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.906975    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v" event={"ID":"ee7d5e68-ac10-426f-8a0a-f5b5d791f875","Type":"ContainerDied","Data":"0ae85f22ce3f6704aab0b3ea3eec3b811ec58ff03462ab0a521a56b7192254b9"}
Nov 03 01:07:22.907047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.907010    4550 scope.go:122] "RemoveContainer" containerID="3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2"
Nov 03 01:07:22.907220 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.907203    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"
Nov 03 01:07:22.921310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.921279    4550 scope.go:122] "RemoveContainer" containerID="27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"
Nov 03 01:07:22.935389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.935361    4550 scope.go:122] "RemoveContainer" containerID="3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2"
Nov 03 01:07:22.935833 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:22.935792    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2\": container with ID starting with 3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2 not found: ID does not exist" containerID="3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2"
Nov 03 01:07:22.935976 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.935850    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2"} err="failed to get container status \"3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2\": rpc error: code = NotFound desc = could not find container \"3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2\": container with ID starting with 3de422d8b06a88e27851f926e75107ecea45c5ddeba8b8d615d58c0f53c829f2 not found: ID does not exist"
Nov 03 01:07:22.935976 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.935876    4550 scope.go:122] "RemoveContainer" containerID="27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"
Nov 03 01:07:22.936229 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:22.936205    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001\": container with ID starting with 27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001 not found: ID does not exist" containerID="27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"
Nov 03 01:07:22.936357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.936232    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001"} err="failed to get container status \"27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001\": rpc error: code = NotFound desc = could not find container \"27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001\": container with ID starting with 27676f57b5b8f8add32f2209414189e8b46f3254967812a58dada70861bbb001 not found: ID does not exist"
Nov 03 01:07:22.948360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.948332    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.992885    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993057    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993139    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993234    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993390    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993500    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993670    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.993950    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.994493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.994031    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket\") pod \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\" (UID: \"ee7d5e68-ac10-426f-8a0a-f5b5d791f875\") "
Nov 03 01:07:22.995710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.995526    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:22.996081 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:22.995878    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:23.004516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.004471    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:23.004867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.004842    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:23.005132 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.005111    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:23.005307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.005286    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:23.019024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.018903    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:07:23.020145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.020117    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "kube-api-access-znzjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:23.020344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.020318    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy" pod "ee7d5e68-ac10-426f-8a0a-f5b5d791f875" (UID: "ee7d5e68-ac10-426f-8a0a-f5b5d791f875"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:23.096478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096390    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096472    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096484    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znzjm\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-kube-api-access-znzjm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096494    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096504    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096512    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096520    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096528    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.096831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.096536    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee7d5e68-ac10-426f-8a0a-f5b5d791f875-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:23.255645 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.255604    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"]
Nov 03 01:07:23.259759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.259727    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8ed55a-openshift-default-55cf654d4c-wl24v"]
Nov 03 01:07:23.812112 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:23.812078    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" path="/var/lib/kubelet/pods/ee7d5e68-ac10-426f-8a0a-f5b5d791f875/volumes"
Nov 03 01:07:24.465492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:24.465453    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"]
Nov 03 01:07:24.487362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:24.487327    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"]
Nov 03 01:07:24.601055 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:24.601014    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:24.688072 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:24.688036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:24.764741 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:24.764634    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:07:24.790010 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:24.789972    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:25.911210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:25.911174    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:26.347494 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:26.347385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:27.226393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:27.226362    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:27.706555 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:27.706452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:28.152464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:28.152385    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"]
Nov 03 01:07:28.252188 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:28.252133    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"]
Nov 03 01:07:28.524156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:28.524028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:28.708281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:28.708235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:29.477829 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:29.477512    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p" podUID="ec65abdf-e24f-4390-a5fb-15b563cde55d" containerName="payload-processing" containerID="cri-o://3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109" gracePeriod=25
Nov 03 01:07:29.502145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:29.490354    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerName="payload-pre-processing" containerID="cri-o://7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51" gracePeriod=25
Nov 03 01:07:29.851897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:29.851855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:29.960453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:29.954534    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 01:07:30.047082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.044906    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume\") pod \"ec65abdf-e24f-4390-a5fb-15b563cde55d\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") "
Nov 03 01:07:30.047082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.045009    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg\" (UniqueName: \"kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg\") pod \"ec65abdf-e24f-4390-a5fb-15b563cde55d\" (UID: \"ec65abdf-e24f-4390-a5fb-15b563cde55d\") "
Nov 03 01:07:30.047082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.046256    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume" pod "ec65abdf-e24f-4390-a5fb-15b563cde55d" (UID: "ec65abdf-e24f-4390-a5fb-15b563cde55d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:30.063875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.063814    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg" pod "ec65abdf-e24f-4390-a5fb-15b563cde55d" (UID: "ec65abdf-e24f-4390-a5fb-15b563cde55d"). InnerVolumeSpecName "kube-api-access-9l9vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:30.146394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.146357    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec65abdf-e24f-4390-a5fb-15b563cde55d-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:30.146394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.146387    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9vg\" (UniqueName: \"kubernetes.io/projected/ec65abdf-e24f-4390-a5fb-15b563cde55d-kube-api-access-9l9vg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:30.227062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.227028    4550 generic.go:356] "Generic (PLEG): container finished" podID="ec65abdf-e24f-4390-a5fb-15b563cde55d" containerID="3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109" exitCode=0
Nov 03 01:07:30.227231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.227077    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p" event={"ID":"ec65abdf-e24f-4390-a5fb-15b563cde55d","Type":"ContainerDied","Data":"3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109"}
Nov 03 01:07:30.227231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.227081    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"
Nov 03 01:07:30.227231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.227120    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p" event={"ID":"ec65abdf-e24f-4390-a5fb-15b563cde55d","Type":"ContainerDied","Data":"ede794e16fcf3abe0f78a5e6cd05b9b44e4a41d197393bd510ff6420e57c14c8"}
Nov 03 01:07:30.227231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.227142    4550 scope.go:122] "RemoveContainer" containerID="3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109"
Nov 03 01:07:30.239011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.238991    4550 scope.go:122] "RemoveContainer" containerID="3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109"
Nov 03 01:07:30.239354 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:30.239330    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109\": container with ID starting with 3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109 not found: ID does not exist" containerID="3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109"
Nov 03 01:07:30.239512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.239361    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109"} err="failed to get container status \"3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109\": rpc error: code = NotFound desc = could not find container \"3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109\": container with ID starting with 3c24b623fdbe1f95e5437975d7107e11878a1833ad5effd15938bac31cd62109 not found: ID does not exist"
Nov 03 01:07:30.283729 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.283697    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"]
Nov 03 01:07:30.289924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.289884    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c7d6-67775dcc4f-nmv5p"]
Nov 03 01:07:30.679226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.679186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:30.836711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:30.836670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:31.340371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:31.340333    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:31.711262 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:31.711219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:31.713563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:31.713531    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"]
Nov 03 01:07:31.713822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:31.713797    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" containerID="cri-o://46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77" gracePeriod=30
Nov 03 01:07:31.875455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:31.871860    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:31.917317 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:31.917277    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec65abdf-e24f-4390-a5fb-15b563cde55d" path="/var/lib/kubelet/pods/ec65abdf-e24f-4390-a5fb-15b563cde55d/volumes"
Nov 03 01:07:32.770604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:32.770564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:32.937349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:32.937308    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:32.941252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:32.941221    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:33.042626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:33.042525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:33.153848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:33.153788    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerName="payload-processing" containerID="cri-o://1572511217e237c2e9d04e1bc8811169a31d33fd7663dd100c18c9cf16f7bb6e" gracePeriod=25
Nov 03 01:07:33.255015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:33.254973    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerName="payload-pre-processing" containerID="cri-o://45699819513af5f41c54eaccb86837d13b4b2b3c97f9b92a4037f17ebca4a0ae" gracePeriod=25
Nov 03 01:07:33.631365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:33.631325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:34.433738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.433686    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:34.844963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.844932    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 01:07:34.913805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.913761    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume\") pod \"6a97e033-437f-4794-aad7-a9454d4e23a4\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") "
Nov 03 01:07:34.914608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.914255    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume" pod "6a97e033-437f-4794-aad7-a9454d4e23a4" (UID: "6a97e033-437f-4794-aad7-a9454d4e23a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:34.914608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.914525    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm\" (UniqueName: \"kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm\") pod \"6a97e033-437f-4794-aad7-a9454d4e23a4\" (UID: \"6a97e033-437f-4794-aad7-a9454d4e23a4\") "
Nov 03 01:07:34.915633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.915233    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a97e033-437f-4794-aad7-a9454d4e23a4-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:34.919415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:34.919347    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm" pod "6a97e033-437f-4794-aad7-a9454d4e23a4" (UID: "6a97e033-437f-4794-aad7-a9454d4e23a4"). InnerVolumeSpecName "kube-api-access-mbpcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:35.016593 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.016492    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbpcm\" (UniqueName: \"kubernetes.io/projected/6a97e033-437f-4794-aad7-a9454d4e23a4-kube-api-access-mbpcm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:35.269559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.269466    4550 generic.go:356] "Generic (PLEG): container finished" podID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerID="2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84" exitCode=0
Nov 03 01:07:35.269559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.269505    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"
Nov 03 01:07:35.269559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.269530    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" event={"ID":"6a97e033-437f-4794-aad7-a9454d4e23a4","Type":"ContainerDied","Data":"2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84"}
Nov 03 01:07:35.269856 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.269602    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh" event={"ID":"6a97e033-437f-4794-aad7-a9454d4e23a4","Type":"ContainerDied","Data":"ca229585579623b9066219184bc6ef905fc54da1dfd115ad14a92ae7fc64509d"}
Nov 03 01:07:35.269856 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.269634    4550 scope.go:122] "RemoveContainer" containerID="2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84"
Nov 03 01:07:35.282791 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.282769    4550 scope.go:122] "RemoveContainer" containerID="2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84"
Nov 03 01:07:35.283227 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:35.283198    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84\": container with ID starting with 2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84 not found: ID does not exist" containerID="2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84"
Nov 03 01:07:35.283368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.283233    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84"} err="failed to get container status \"2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84\": rpc error: code = NotFound desc = could not find container \"2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84\": container with ID starting with 2f7ba9ff6166435e6151bd43b3938c77bf0afe0edbcd9fb99b63da59271bae84 not found: ID does not exist"
Nov 03 01:07:35.306014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.305982    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:35.324918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.324881    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"]
Nov 03 01:07:35.328841 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.328809    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8ed55a-db6c776cf-vdsjh"]
Nov 03 01:07:35.514023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.513982    4550 patch_prober.go:29] interesting pod/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:07:35.514483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.514063    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:07:35.810668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:35.810631    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" path="/var/lib/kubelet/pods/6a97e033-437f-4794-aad7-a9454d4e23a4/volumes"
Nov 03 01:07:36.933471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:36.933396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:37.039293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.039263    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 01:07:37.144886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.144851    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.144918    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.144968    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145005    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145061    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145114    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145222    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145286    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:37.145329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145261    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145375    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:37.145608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145449    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket\") pod \"4d304809-13c7-4fff-8287-b781199a7e3d\" (UID: \"4d304809-13c7-4fff-8287-b781199a7e3d\") "
Nov 03 01:07:37.145608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145538    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:37.145775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145674    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:37.145817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.145790    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:37.146159 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.146136    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.146199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.146166    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.146199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.146181    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d304809-13c7-4fff-8287-b781199a7e3d-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.146292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.146196    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.146292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.146210    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.147953 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.147927    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:07:37.148230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.148209    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:37.148469 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.148426    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "kube-api-access-fdv7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:37.148778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.148761    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy" pod "4d304809-13c7-4fff-8287-b781199a7e3d" (UID: "4d304809-13c7-4fff-8287-b781199a7e3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:37.248078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.247987    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d304809-13c7-4fff-8287-b781199a7e3d-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.248078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.248021    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d304809-13c7-4fff-8287-b781199a7e3d-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.248078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.248031    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdv7t\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-kube-api-access-fdv7t\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.248078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.248040    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d304809-13c7-4fff-8287-b781199a7e3d-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.297879 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.297836    4550 generic.go:356] "Generic (PLEG): container finished" podID="4d304809-13c7-4fff-8287-b781199a7e3d" containerID="46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77" exitCode=0
Nov 03 01:07:37.298183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.298150    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" event={"ID":"4d304809-13c7-4fff-8287-b781199a7e3d","Type":"ContainerDied","Data":"46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77"}
Nov 03 01:07:37.298363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.298246    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266" event={"ID":"4d304809-13c7-4fff-8287-b781199a7e3d","Type":"ContainerDied","Data":"619ab0fd8c284010821116f6b5a20eabc5d44c3f6bbfc3fdf1c82d8a7c06c5c1"}
Nov 03 01:07:37.298363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.298290    4550 scope.go:122] "RemoveContainer" containerID="46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77"
Nov 03 01:07:37.298672 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.298545    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"
Nov 03 01:07:37.312711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.312690    4550 scope.go:122] "RemoveContainer" containerID="713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"
Nov 03 01:07:37.323416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.323384    4550 scope.go:122] "RemoveContainer" containerID="46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77"
Nov 03 01:07:37.323807 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:37.323782    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77\": container with ID starting with 46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77 not found: ID does not exist" containerID="46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77"
Nov 03 01:07:37.323859 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.323818    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77"} err="failed to get container status \"46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77\": rpc error: code = NotFound desc = could not find container \"46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77\": container with ID starting with 46d9983efa05ea66aef8f4cdfa91030238cc573d1c29ebbcbc1e859839edcc77 not found: ID does not exist"
Nov 03 01:07:37.323859 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.323838    4550 scope.go:122] "RemoveContainer" containerID="713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"
Nov 03 01:07:37.324115 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:37.324094    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850\": container with ID starting with 713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850 not found: ID does not exist" containerID="713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"
Nov 03 01:07:37.324166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.324113    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850"} err="failed to get container status \"713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850\": rpc error: code = NotFound desc = could not find container \"713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850\": container with ID starting with 713be82b441840ccf8214e65d122fe4a3a38a419182af8902f52fb2912eb4850 not found: ID does not exist"
Nov 03 01:07:37.354671 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.354635    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"]
Nov 03 01:07:37.358467 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.358406    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-70c7d6-openshift-default-7f7f5bb95-57266"]
Nov 03 01:07:37.669654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.669609    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:37.810054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:37.810006    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" path="/var/lib/kubelet/pods/4d304809-13c7-4fff-8287-b781199a7e3d/volumes"
Nov 03 01:07:38.722391 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:38.722352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:39.133679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:39.133590    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:39.234225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:39.234192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:39.744197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:39.744151    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:40.386969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:40.386932    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:40.920630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:40.920584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:42.038670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:42.038633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:42.939876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:42.939833    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:43.727392 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:43.727353    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:45.335110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:45.335064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:46.944168 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:46.944126    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:47.669108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:47.669069    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:48.626412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:48.626373    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:49.743680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.743648    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 01:07:49.744424 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.744398    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:49.839543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.839447    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp\" (UniqueName: \"kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp\") pod \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") "
Nov 03 01:07:49.839730 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.839718    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume\") pod \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\" (UID: \"50aa2c23-760a-4be5-82b9-bbbd90b4ad64\") "
Nov 03 01:07:49.840094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.840072    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume" pod "50aa2c23-760a-4be5-82b9-bbbd90b4ad64" (UID: "50aa2c23-760a-4be5-82b9-bbbd90b4ad64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:49.841579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.841555    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp" pod "50aa2c23-760a-4be5-82b9-bbbd90b4ad64" (UID: "50aa2c23-760a-4be5-82b9-bbbd90b4ad64"). InnerVolumeSpecName "kube-api-access-rhqtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:49.941851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.941812    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqtp\" (UniqueName: \"kubernetes.io/projected/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-kube-api-access-rhqtp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:49.941851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:49.941846    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50aa2c23-760a-4be5-82b9-bbbd90b4ad64-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:50.028872 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.028828    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"]
Nov 03 01:07:50.032646 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.032603    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a0feefd-hrzds"]
Nov 03 01:07:50.211300 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.211245    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:50.404980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.404939    4550 generic.go:356] "Generic (PLEG): container finished" podID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerID="7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51" exitCode=0
Nov 03 01:07:50.405207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.404987    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"
Nov 03 01:07:50.405207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.405021    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" event={"ID":"50aa2c23-760a-4be5-82b9-bbbd90b4ad64","Type":"ContainerDied","Data":"7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51"}
Nov 03 01:07:50.405207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.405086    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx" event={"ID":"50aa2c23-760a-4be5-82b9-bbbd90b4ad64","Type":"ContainerDied","Data":"815fbf5172fa0f00658024bd3128de4436b572b4f11f36394f4e8b816cde32b9"}
Nov 03 01:07:50.405207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.405121    4550 scope.go:122] "RemoveContainer" containerID="7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51"
Nov 03 01:07:50.416698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.416671    4550 scope.go:122] "RemoveContainer" containerID="7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51"
Nov 03 01:07:50.417074 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:50.417050    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51\": container with ID starting with 7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51 not found: ID does not exist" containerID="7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51"
Nov 03 01:07:50.417147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.417089    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51"} err="failed to get container status \"7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51\": rpc error: code = NotFound desc = could not find container \"7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51\": container with ID starting with 7ed419e881cb4299907c814501a27c0f712af1899b11b58a9d398cccb0b55a51 not found: ID does not exist"
Nov 03 01:07:50.460837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.460803    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"]
Nov 03 01:07:50.474356 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:50.474253    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-70c7d6-7887bfd775-2lqfx"]
Nov 03 01:07:51.811024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:51.810979    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" path="/var/lib/kubelet/pods/50aa2c23-760a-4be5-82b9-bbbd90b4ad64/volumes"
Nov 03 01:07:51.811701 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:51.811679    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d68cef5-adeb-4b15-80f8-6828b5caba42" path="/var/lib/kubelet/pods/6d68cef5-adeb-4b15-80f8-6828b5caba42/volumes"
Nov 03 01:07:51.847897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:51.847860    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:53.442532 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.442494    4550 generic.go:356] "Generic (PLEG): container finished" podID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerID="45699819513af5f41c54eaccb86837d13b4b2b3c97f9b92a4037f17ebca4a0ae" exitCode=0
Nov 03 01:07:53.442960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.442542    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" event={"ID":"ec2d13d0-101e-405d-8ca5-87e709269cfc","Type":"ContainerDied","Data":"45699819513af5f41c54eaccb86837d13b4b2b3c97f9b92a4037f17ebca4a0ae"}
Nov 03 01:07:53.451739 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.451560    4550 generic.go:356] "Generic (PLEG): container finished" podID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerID="1572511217e237c2e9d04e1bc8811169a31d33fd7663dd100c18c9cf16f7bb6e" exitCode=0
Nov 03 01:07:53.451739 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.451641    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" event={"ID":"dfda3998-0005-44bb-9dad-f09c1ae69178","Type":"ContainerDied","Data":"1572511217e237c2e9d04e1bc8811169a31d33fd7663dd100c18c9cf16f7bb6e"}
Nov 03 01:07:53.466614 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.466583    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 01:07:53.501375 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.494916    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume\") pod \"dfda3998-0005-44bb-9dad-f09c1ae69178\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") "
Nov 03 01:07:53.501375 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.495052    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv\" (UniqueName: \"kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv\") pod \"dfda3998-0005-44bb-9dad-f09c1ae69178\" (UID: \"dfda3998-0005-44bb-9dad-f09c1ae69178\") "
Nov 03 01:07:53.502505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.502424    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume" pod "dfda3998-0005-44bb-9dad-f09c1ae69178" (UID: "dfda3998-0005-44bb-9dad-f09c1ae69178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:53.511126 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.511066    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv" pod "dfda3998-0005-44bb-9dad-f09c1ae69178" (UID: "dfda3998-0005-44bb-9dad-f09c1ae69178"). InnerVolumeSpecName "kube-api-access-lrwkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:53.547811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.546323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:53.565637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.565602    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 01:07:53.612860 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.612798    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5\" (UniqueName: \"kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5\") pod \"ec2d13d0-101e-405d-8ca5-87e709269cfc\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") "
Nov 03 01:07:53.625227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.625186    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume\") pod \"ec2d13d0-101e-405d-8ca5-87e709269cfc\" (UID: \"ec2d13d0-101e-405d-8ca5-87e709269cfc\") "
Nov 03 01:07:53.626114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.626084    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfda3998-0005-44bb-9dad-f09c1ae69178-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:53.626512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.626477    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrwkv\" (UniqueName: \"kubernetes.io/projected/dfda3998-0005-44bb-9dad-f09c1ae69178-kube-api-access-lrwkv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:53.627147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.627121    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume" pod "ec2d13d0-101e-405d-8ca5-87e709269cfc" (UID: "ec2d13d0-101e-405d-8ca5-87e709269cfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:53.647419 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.647378    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5" pod "ec2d13d0-101e-405d-8ca5-87e709269cfc" (UID: "ec2d13d0-101e-405d-8ca5-87e709269cfc"). InnerVolumeSpecName "kube-api-access-v65n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:53.728603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.728511    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v65n5\" (UniqueName: \"kubernetes.io/projected/ec2d13d0-101e-405d-8ca5-87e709269cfc-kube-api-access-v65n5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:53.728603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:53.728543    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec2d13d0-101e-405d-8ca5-87e709269cfc-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:07:54.462575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.462472    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg" event={"ID":"dfda3998-0005-44bb-9dad-f09c1ae69178","Type":"ContainerDied","Data":"9c782bc00d64456f4e116d6e950f6df556cece327ae36c33347de8d3a618d95c"}
Nov 03 01:07:54.462575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.462565    4550 scope.go:122] "RemoveContainer" containerID="1572511217e237c2e9d04e1bc8811169a31d33fd7663dd100c18c9cf16f7bb6e"
Nov 03 01:07:54.463139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.462485    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"
Nov 03 01:07:54.465081 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.464989    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"
Nov 03 01:07:54.465081 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.464996    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4" event={"ID":"ec2d13d0-101e-405d-8ca5-87e709269cfc","Type":"ContainerDied","Data":"d526d674764d88c8c813846a168b68483ca87c87f530dc1b585568f053a822d7"}
Nov 03 01:07:54.478632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.478597    4550 scope.go:122] "RemoveContainer" containerID="45699819513af5f41c54eaccb86837d13b4b2b3c97f9b92a4037f17ebca4a0ae"
Nov 03 01:07:54.516832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.516799    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"]
Nov 03 01:07:54.529994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.529962    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-537652-7d5c567859-n28j4"]
Nov 03 01:07:54.555728 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.555691    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"]
Nov 03 01:07:54.561130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:54.561096    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-537652-5dd9974447-tfbxg"]
Nov 03 01:07:54.731540 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:07:54.731387    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:07:55.224775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:55.224739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:55.809515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:55.809479    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" path="/var/lib/kubelet/pods/dfda3998-0005-44bb-9dad-f09c1ae69178/volumes"
Nov 03 01:07:55.810027 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:55.810010    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" path="/var/lib/kubelet/pods/ec2d13d0-101e-405d-8ca5-87e709269cfc/volumes"
Nov 03 01:07:56.925027 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:56.924993    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:58.477926 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:58.477884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:07:59.912227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:07:59.912188    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:00.244697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.244603    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:00.322621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.322582    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"]
Nov 03 01:08:00.323331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323279    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerName="payload-pre-processing"
Nov 03 01:08:00.323538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323497    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerName="payload-pre-processing"
Nov 03 01:08:00.323651 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323636    4550 cpu_manager.go:414] "containerMap: removing container" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerName="payload-pre-processing"
Nov 03 01:08:00.323741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323651    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerName="payload-pre-processing"
Nov 03 01:08:00.323741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323696    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy"
Nov 03 01:08:00.323741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323706    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy"
Nov 03 01:08:00.323741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323718    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerName="payload-processing"
Nov 03 01:08:00.323741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323725    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerName="payload-processing"
Nov 03 01:08:00.323741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323737    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323744    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323754    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerName="payload-pre-processing"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323761    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerName="payload-pre-processing"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323824    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ec65abdf-e24f-4390-a5fb-15b563cde55d" containerName="payload-processing"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323832    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec65abdf-e24f-4390-a5fb-15b563cde55d" containerName="payload-processing"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323849    4550 cpu_manager.go:414] "containerMap: removing container" podUID="33e08195-7ad1-4b3f-9a8d-d57288aea242" containerName="revoke-keys"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323856    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e08195-7ad1-4b3f-9a8d-d57288aea242" containerName="revoke-keys"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323868    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323875    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323888    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.323894    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324081    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324097    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec65abdf-e24f-4390-a5fb-15b563cde55d" containerName="payload-processing"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324108    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e08195-7ad1-4b3f-9a8d-d57288aea242" containerName="revoke-keys"
Nov 03 01:08:00.324114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324118    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy"
Nov 03 01:08:00.324894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324129    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7d5e68-ac10-426f-8a0a-f5b5d791f875" containerName="istio-proxy"
Nov 03 01:08:00.324894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324140    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a97e033-437f-4794-aad7-a9454d4e23a4" containerName="payload-pre-processing"
Nov 03 01:08:00.324894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324153    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="50aa2c23-760a-4be5-82b9-bbbd90b4ad64" containerName="payload-pre-processing"
Nov 03 01:08:00.324894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324166    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd9175e-57ee-4fe2-9cfb-1e3b2eb4faac" containerName="istio-proxy"
Nov 03 01:08:00.324894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324175    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfda3998-0005-44bb-9dad-f09c1ae69178" containerName="payload-processing"
Nov 03 01:08:00.324894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.324185    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2d13d0-101e-405d-8ca5-87e709269cfc" containerName="payload-pre-processing"
Nov 03 01:08:00.377922 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.377879    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"]
Nov 03 01:08:00.378157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.378012    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.384870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.384841    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:00.399200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.399169    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c6e808d7-openshift-default-dockercfg-787vs\""
Nov 03 01:08:00.609555 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.560547    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.693100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.693060    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxc9\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.693378 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.693361    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.699465 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.693558    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.706586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.699912    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.738753 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.680866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:00.743348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.742571    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.743348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.742658    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.743348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.742685    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.743348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.742721    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844358    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844414    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxc9\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844450    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844470    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844559    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844590    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.844655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844645    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844668    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.844700    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.845069    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.845271    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.845357    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.845369    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.845587 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.845563    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.847740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.847709    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.847740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.847710    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.865851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.865769    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxc9\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:00.866162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:00.865914    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token\") pod \"e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") " pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:01.007052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.007013    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:01.341712 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:08:01.341669    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55fb772_8e34_46c9_848d_508b2c44d857.slice/crio-49616f0d02712e5c4a93b0b77fa97148ec67f5517fb7762097cd0fb5e23344a6 WatchSource:0}: Error finding container 49616f0d02712e5c4a93b0b77fa97148ec67f5517fb7762097cd0fb5e23344a6: Status 404 returned error can't find the container with id 49616f0d02712e5c4a93b0b77fa97148ec67f5517fb7762097cd0fb5e23344a6
Nov 03 01:08:01.341867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.341818    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"]
Nov 03 01:08:01.345473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.345404    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:08:01.345598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.345537    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:08:01.345598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.345584    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:08:01.742455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.741596    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" event={"ID":"a55fb772-8e34-46c9-848d-508b2c44d857","Type":"ContainerStarted","Data":"04090e33e1f66e6cc4c52d5a0fd4f8691cae2e58f62a43cab77fc2e537274fc8"}
Nov 03 01:08:01.742455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.741665    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" event={"ID":"a55fb772-8e34-46c9-848d-508b2c44d857","Type":"ContainerStarted","Data":"49616f0d02712e5c4a93b0b77fa97148ec67f5517fb7762097cd0fb5e23344a6"}
Nov 03 01:08:01.924745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:01.924710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:02.008156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:02.008048    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:02.030175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:02.030131    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:02.076275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:02.076217    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podStartSLOduration=2.076203238 podStartE2EDuration="2.076203238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:01.799934636 +0000 UTC m=+3764.121471437" watchObservedRunningTime="2026-11-03 01:08:02.076203238 +0000 UTC m=+3764.397740038"
Nov 03 01:08:02.750273 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:02.750240    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:02.751345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:02.751326    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:08:03.666942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:03.666901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:04.229274 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:04.229236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:04.327256 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:04.327217    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:05.530160 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:05.530111    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:05.878297 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:05.878198    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:05.987243 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:05.987207    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:06.170511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:06.161576    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:07.350785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.350745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:07.488934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.488897    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"]
Nov 03 01:08:07.489988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.489963    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy"
Nov 03 01:08:07.489988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.489987    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy"
Nov 03 01:08:07.490147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.490136    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d304809-13c7-4fff-8287-b781199a7e3d" containerName="istio-proxy"
Nov 03 01:08:07.494594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.494553    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.511541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.511497    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"]
Nov 03 01:08:07.518485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.518458    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"default-dockercfg-d8kkj\""
Nov 03 01:08:07.518830 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.518792    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"rate-test-model-20109d-kserve-self-signed-certs\""
Nov 03 01:08:07.524030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.524001    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:07.524793 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.524771    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"openshift-service-ca.crt\""
Nov 03 01:08:07.524909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.524793    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"kube-root-ca.crt\""
Nov 03 01:08:07.564748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.564712    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.565065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.565043    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.565183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.565158    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.565240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.565225    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.565288 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.565258    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4pw7\" (UniqueName: \"kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.565421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.565402    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.667073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.667032    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.667303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.667100    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.667303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.667142    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.667303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.667163    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4pw7\" (UniqueName: \"kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.667303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.667218    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.667303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.667241    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.668135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.668103    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.668425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.668399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.668771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.668744    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.670091 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.670065    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.670855 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.670830    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.688908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.688872    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4pw7\" (UniqueName: \"kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7\") pod \"rate-test-model-20109d-kserve-6cf886b8df-pkwkv\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") " pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:07.830455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:07.828608    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:08.208045 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.208016    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"]
Nov 03 01:08:08.323506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.323468    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"]
Nov 03 01:08:08.328154 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.327922    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.346474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.343708    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:08:08.346474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.343730    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:08:08.350075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.350044    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"]
Nov 03 01:08:08.385343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.385299    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc64n\" (UniqueName: \"kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.385879 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.385370    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.385879 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.385541    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.487125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.487019    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.487304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.487268    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc64n\" (UniqueName: \"kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.487380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.487313    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.487754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.487730    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.489825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.489804    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.509333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.509284    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc64n\" (UniqueName: \"kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n\") pod \"maas-api-revoke-sub-2a6c39b0-8xns6\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.642971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.642928    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:08.890511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.889767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:08.890511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.890395    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" event={"ID":"a6605688-10aa-4a3f-9601-283f0350e94a","Type":"ContainerStarted","Data":"01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c"}
Nov 03 01:08:08.890511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.890474    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" event={"ID":"a6605688-10aa-4a3f-9601-283f0350e94a","Type":"ContainerStarted","Data":"f6c0edfcbf99ff0fd3f03656b9d2071d06791489b552e033d4aae2a0340fc3d3"}
Nov 03 01:08:08.938923 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.938869    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" podStartSLOduration=1.9388534160000002 podStartE2EDuration="1.938853416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:08.937511878 +0000 UTC m=+3771.259048674" watchObservedRunningTime="2026-11-03 01:08:08.938853416 +0000 UTC m=+3771.260390215"
Nov 03 01:08:08.976955 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:08.976905    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"]
Nov 03 01:08:08.977708 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:08:08.977606    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af6a7b0_b2eb_490b_b888_3453657ed551.slice/crio-86b06d4efa54c7be3299115ab23875293fe4aac2397d9e098291ff6d382b4b57 WatchSource:0}: Error finding container 86b06d4efa54c7be3299115ab23875293fe4aac2397d9e098291ff6d382b4b57: Status 404 returned error can't find the container with id 86b06d4efa54c7be3299115ab23875293fe4aac2397d9e098291ff6d382b4b57
Nov 03 01:08:09.657360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:09.657314    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:09.830096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:09.830003    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:09.901145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:09.901092    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6" event={"ID":"0af6a7b0-b2eb-490b-b888-3453657ed551","Type":"ContainerStarted","Data":"31410ddd7d4d2e663a11429f55f325241dd4d6e817bf118ef69bb3b17f0d287e"}
Nov 03 01:08:09.901351 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:09.901165    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6" event={"ID":"0af6a7b0-b2eb-490b-b888-3453657ed551","Type":"ContainerStarted","Data":"86b06d4efa54c7be3299115ab23875293fe4aac2397d9e098291ff6d382b4b57"}
Nov 03 01:08:09.946306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:09.946235    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6" podStartSLOduration=1.9462202400000002 podStartE2EDuration="1.94622024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:09.943379358 +0000 UTC m=+3772.264916161" watchObservedRunningTime="2026-11-03 01:08:09.94622024 +0000 UTC m=+3772.267757040"
Nov 03 01:08:10.909924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:10.909884    4550 generic.go:356] "Generic (PLEG): container finished" podID="0af6a7b0-b2eb-490b-b888-3453657ed551" containerID="31410ddd7d4d2e663a11429f55f325241dd4d6e817bf118ef69bb3b17f0d287e" exitCode=0
Nov 03 01:08:10.910405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:10.909977    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6" event={"ID":"0af6a7b0-b2eb-490b-b888-3453657ed551","Type":"ContainerDied","Data":"31410ddd7d4d2e663a11429f55f325241dd4d6e817bf118ef69bb3b17f0d287e"}
Nov 03 01:08:11.381112 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:11.381019    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:12.106210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.106185    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:12.145108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.145074    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca\") pod \"0af6a7b0-b2eb-490b-b888-3453657ed551\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") "
Nov 03 01:08:12.145504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.145472    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca\") pod \"0af6a7b0-b2eb-490b-b888-3453657ed551\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") "
Nov 03 01:08:12.145655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.145536    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca" pod "0af6a7b0-b2eb-490b-b888-3453657ed551" (UID: "0af6a7b0-b2eb-490b-b888-3453657ed551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:08:12.145743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.145648    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n\" (UniqueName: \"kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n\") pod \"0af6a7b0-b2eb-490b-b888-3453657ed551\" (UID: \"0af6a7b0-b2eb-490b-b888-3453657ed551\") "
Nov 03 01:08:12.146349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.146318    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:12.148390 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.148354    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca" pod "0af6a7b0-b2eb-490b-b888-3453657ed551" (UID: "0af6a7b0-b2eb-490b-b888-3453657ed551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:08:12.148666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.148641    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n" pod "0af6a7b0-b2eb-490b-b888-3453657ed551" (UID: "0af6a7b0-b2eb-490b-b888-3453657ed551"). InnerVolumeSpecName "kube-api-access-tc64n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:08:12.247491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.247401    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc64n\" (UniqueName: \"kubernetes.io/projected/0af6a7b0-b2eb-490b-b888-3453657ed551-kube-api-access-tc64n\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:12.247491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.247466    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af6a7b0-b2eb-490b-b888-3453657ed551-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:12.926849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.926819    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"
Nov 03 01:08:12.927017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.926859    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6" event={"ID":"0af6a7b0-b2eb-490b-b888-3453657ed551","Type":"ContainerDied","Data":"86b06d4efa54c7be3299115ab23875293fe4aac2397d9e098291ff6d382b4b57"}
Nov 03 01:08:12.927017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.926916    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86b06d4efa54c7be3299115ab23875293fe4aac2397d9e098291ff6d382b4b57"
Nov 03 01:08:12.930097 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:12.930071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:14.638598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:14.638559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:15.176902 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:15.176864    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:16.211814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:16.211777    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:16.318919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:16.318881    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:16.407174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:16.407140    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:16.947604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:16.947564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:17.319275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:17.319176    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"]
Nov 03 01:08:17.320074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:17.320042    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" containerID="cri-o://bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02" gracePeriod=30
Nov 03 01:08:17.329583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:17.329554    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:17.829788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:17.829754    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:17.843242 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:17.843212    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:17.974017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:17.973983    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:08:18.385380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:18.385339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:18.735774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:18.735739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:19.245654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:19.245610    4550 patch_prober.go:29] interesting pod/e2e-inf-b-537652-openshift-default-746478548f-zvwjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:08:19.245843 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:19.245677    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:08:20.224786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:20.224745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:21.889940 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:21.889894    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:22.623074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.623042    4550 scope.go:122] "RemoveContainer" containerID="ce701d4ad0fe7a60fd290e10f34560f442e2c3aaa49638b3940f937ea0af183c"
Nov 03 01:08:22.628846 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.628754    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 01:08:22.724601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724568    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.724799 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724651    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.724799 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724685    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.724799 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724794    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.724973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724833    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.724973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724886    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:08:22.724973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724928    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.725140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.724978    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.725140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.725103    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:08:22.725261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.725133    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:08:22.725671 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.725646    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.725788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.725675    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa828ae-d375-4578-9485-71ed468a41e1-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.725788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.725691    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.727189 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.727165    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "kube-api-access-zwvrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:08:22.727377 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.727357    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:08:22.727760 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.727730    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:08:22.727930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.727915    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:08:22.826560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.826527    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.826780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.826571    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs\") pod \"faa828ae-d375-4578-9485-71ed468a41e1\" (UID: \"faa828ae-d375-4578-9485-71ed468a41e1\") "
Nov 03 01:08:22.826895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.826869    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:08:22.826957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.826885    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs" pod "faa828ae-d375-4578-9485-71ed468a41e1" (UID: "faa828ae-d375-4578-9485-71ed468a41e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:08:22.827164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.827144    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa828ae-d375-4578-9485-71ed468a41e1-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.827226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.827169    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.827226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.827184    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.827226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.827196    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.827226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.827209    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvrj\" (UniqueName: \"kubernetes.io/projected/faa828ae-d375-4578-9485-71ed468a41e1-kube-api-access-zwvrj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:22.827226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:22.827224    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa828ae-d375-4578-9485-71ed468a41e1-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:08:23.005675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.005637    4550 generic.go:356] "Generic (PLEG): container finished" podID="faa828ae-d375-4578-9485-71ed468a41e1" containerID="bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02" exitCode=0
Nov 03 01:08:23.006259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.005689    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"
Nov 03 01:08:23.006259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.005690    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" event={"ID":"faa828ae-d375-4578-9485-71ed468a41e1","Type":"ContainerDied","Data":"bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02"}
Nov 03 01:08:23.006259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.005799    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz" event={"ID":"faa828ae-d375-4578-9485-71ed468a41e1","Type":"ContainerDied","Data":"3a2951c014fc183f0a2404297a8d917fd1b7496f11a9cf839d0bcddb994e9c97"}
Nov 03 01:08:23.006259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.005826    4550 scope.go:122] "RemoveContainer" containerID="bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02"
Nov 03 01:08:23.017308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.017279    4550 scope.go:122] "RemoveContainer" containerID="bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02"
Nov 03 01:08:23.017656 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:08:23.017635    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02\": container with ID starting with bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02 not found: ID does not exist" containerID="bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02"
Nov 03 01:08:23.017727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.017666    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02"} err="failed to get container status \"bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02\": rpc error: code = NotFound desc = could not find container \"bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02\": container with ID starting with bf0f7266f04d4394eb8eb73a9c4f777a38664f44fadee7a7a4716d8447b5ae02 not found: ID does not exist"
Nov 03 01:08:23.088366 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.088325    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"]
Nov 03 01:08:23.092174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.092146    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-537652-openshift-default-746478548f-zvwjz"]
Nov 03 01:08:23.538474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.538416    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:23.812301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:23.812216    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa828ae-d375-4578-9485-71ed468a41e1" path="/var/lib/kubelet/pods/faa828ae-d375-4578-9485-71ed468a41e1/volumes"
Nov 03 01:08:24.730991 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:08:24.730945    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:08:24.922659 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:24.922620    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:25.291121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:25.291061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:26.548813 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:26.548763    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:26.953134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:26.953095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:28.537407 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:28.537367    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:30.230911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:30.230875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:31.765475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:31.765395    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:31.895718 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:31.895674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:33.669113 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:33.669078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:34.300977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:34.300931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:35.338817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:35.338783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:37.164890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:37.164831    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:38.869733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:38.869693    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:40.958298 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:40.958238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:42.831237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:42.831195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:44.570592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:44.570551    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:46.359801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:46.359760    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:48.141469 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:48.141405    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:49.030159 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:49.030121    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"]
Nov 03 01:08:49.042560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:49.042527    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5359e2a7-k6djn"]
Nov 03 01:08:49.815265 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:49.815228    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ce451b-db6e-4107-9842-514f4b20328b" path="/var/lib/kubelet/pods/92ce451b-db6e-4107-9842-514f4b20328b/volumes"
Nov 03 01:08:49.963354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:49.963310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:50.073047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:50.072951    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:51.820375 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.820338    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"]
Nov 03 01:08:51.820957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.820942    4550 cpu_manager.go:414] "containerMap: removing container" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy"
Nov 03 01:08:51.821008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.820958    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy"
Nov 03 01:08:51.821008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.820997    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0af6a7b0-b2eb-490b-b888-3453657ed551" containerName="revoke-keys"
Nov 03 01:08:51.821008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.821003    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af6a7b0-b2eb-490b-b888-3453657ed551" containerName="revoke-keys"
Nov 03 01:08:51.821109 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.821093    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy"
Nov 03 01:08:51.821109 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.821104    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy"
Nov 03 01:08:51.821180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.821115    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af6a7b0-b2eb-490b-b888-3453657ed551" containerName="revoke-keys"
Nov 03 01:08:51.826495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.826327    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:51.846862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.846834    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"openshift-service-ca.crt\""
Nov 03 01:08:51.847201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.847182    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"kube-root-ca.crt\""
Nov 03 01:08:51.859750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.859703    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"default-dockercfg-s2tsc\""
Nov 03 01:08:51.860020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.859999    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"rate-test-model-7ef1b6-kserve-self-signed-certs\""
Nov 03 01:08:51.864724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.864688    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"]
Nov 03 01:08:51.888450 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.888404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:51.952145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.952096    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:51.952370 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.952235    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:51.952370 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.952297    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:51.952555 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.952540    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:51.952619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.952576    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:51.952680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:51.952630    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cq2v\" (UniqueName: \"kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.006870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.006828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.056330    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.056391    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.056468    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cq2v\" (UniqueName: \"kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.056513    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.056582    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.056614    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.057263    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.061822    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.065455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.062111    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.074485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.073760    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.090451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.077777    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.105459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.104164    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cq2v\" (UniqueName: \"kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v\") pod \"rate-test-model-7ef1b6-kserve-69b64885cc-48jxw\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.152454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.151793    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:52.530369 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:08:52.530338    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d4e9365_33fb_4342_8e09_1923fde1c424.slice/crio-d83f6c13504b4dd0dd90a507a7f1a06c767620ed6fa7d4b87f64a33441ca5156 WatchSource:0}: Error finding container d83f6c13504b4dd0dd90a507a7f1a06c767620ed6fa7d4b87f64a33441ca5156: Status 404 returned error can't find the container with id d83f6c13504b4dd0dd90a507a7f1a06c767620ed6fa7d4b87f64a33441ca5156
Nov 03 01:08:52.533023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:52.532999    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"]
Nov 03 01:08:53.330934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:53.330895    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" event={"ID":"6d4e9365-33fb-4342-8e09-1923fde1c424","Type":"ContainerStarted","Data":"79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4"}
Nov 03 01:08:53.331360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:53.330945    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" event={"ID":"6d4e9365-33fb-4342-8e09-1923fde1c424","Type":"ContainerStarted","Data":"d83f6c13504b4dd0dd90a507a7f1a06c767620ed6fa7d4b87f64a33441ca5156"}
Nov 03 01:08:53.377110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:53.377055    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" podStartSLOduration=2.377040093 podStartE2EDuration="2.377040093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:53.376657545 +0000 UTC m=+3815.698194344" watchObservedRunningTime="2026-11-03 01:08:53.377040093 +0000 UTC m=+3815.698576897"
Nov 03 01:08:53.657867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:53.657817    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:54.153210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:54.153171    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:08:54.725323 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:08:54.725285    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:08:55.205353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:55.205315    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:55.396252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:55.396205    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:57.059195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:57.059144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:08:58.825588 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:08:58.825551    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:00.035896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:00.035812    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"]
Nov 03 01:09:00.049086 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:00.049054    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc13d928-kkkpx"]
Nov 03 01:09:00.619241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:00.619202    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:01.809021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:01.808986    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07b257b-661d-4b2d-8d11-47cfc653efd3" path="/var/lib/kubelet/pods/c07b257b-661d-4b2d-8d11-47cfc653efd3/volumes"
Nov 03 01:09:02.153490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:02.153460    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:09:02.166506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:02.166474    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:09:02.345796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:02.345743    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:02.443852 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:02.443758    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:09:04.054929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:04.054896    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:05.731884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:05.731844    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:07.410894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:07.410861    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:07.781768 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:07.781674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:09.078713 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:09.078674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:10.683924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:10.683851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:12.414608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:12.414569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:14.056281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:14.056235    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:15.658253 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:15.658216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:16.033412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:16.033304    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"]
Nov 03 01:09:16.046352 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:16.046312    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e668813-vq5x4"]
Nov 03 01:09:16.146125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:16.146083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:17.809362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:17.809317    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7320a705-f3f3-4aa3-af4a-15972677bc0b" path="/var/lib/kubelet/pods/7320a705-f3f3-4aa3-af4a-15972677bc0b/volumes"
Nov 03 01:09:18.004700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:18.004665    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:19.746402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:19.746365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:20.413629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:20.413596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:21.349516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:21.349469    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:22.693045 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:22.693007    4550 scope.go:122] "RemoveContainer" containerID="1dc6af3f4347a8c96a3712f36f83f06b8c6a054d2934db4db97ffd699a503ed0"
Nov 03 01:09:23.013343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:23.013256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:24.583235 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:24.583195    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:24.725305 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:09:24.725271    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:09:26.165491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:26.165425    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:27.932495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:27.932455    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:29.736070 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:29.736038    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:31.388164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:31.388129    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:33.136786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:33.136745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:34.929208 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:34.929149    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:36.594131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:36.594046    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:38.443798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:38.443753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:40.130263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:40.130229    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:41.237830 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:41.237791    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:41.979755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:41.979706    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:42.170891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:42.170855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:42.290445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:42.290332    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:42.804875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:42.804839    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:43.817927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:43.817887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:44.780749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:44.780712    4550 generic.go:356] "Generic (PLEG): container finished" podID="a55fb772-8e34-46c9-848d-508b2c44d857" containerID="04090e33e1f66e6cc4c52d5a0fd4f8691cae2e58f62a43cab77fc2e537274fc8" exitCode=0
Nov 03 01:09:44.780997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:44.780786    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" event={"ID":"a55fb772-8e34-46c9-848d-508b2c44d857","Type":"ContainerDied","Data":"04090e33e1f66e6cc4c52d5a0fd4f8691cae2e58f62a43cab77fc2e537274fc8"}
Nov 03 01:09:44.781284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:44.781265    4550 scope.go:122] "RemoveContainer" containerID="04090e33e1f66e6cc4c52d5a0fd4f8691cae2e58f62a43cab77fc2e537274fc8"
Nov 03 01:09:44.783281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:44.783251    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:09:44.783403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:44.783317    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:09:44.783403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:44.783351    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:09:45.029235 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:45.029192    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"]
Nov 03 01:09:45.032849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:45.032771    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe6f97b5-7lrj8"]
Nov 03 01:09:45.530176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:45.530131    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:45.796771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:45.796678    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" event={"ID":"a55fb772-8e34-46c9-848d-508b2c44d857","Type":"ContainerStarted","Data":"6ab12a6e50ce9ad382b7c464b96ce1dc5d47e96092ebdb287457e2fdb5737823"}
Nov 03 01:09:45.808503 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:45.808472    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04bba05-aee0-497f-b36f-4f91659da630" path="/var/lib/kubelet/pods/c04bba05-aee0-497f-b36f-4f91659da630/volumes"
Nov 03 01:09:46.007981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:46.007937    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:09:46.009592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:46.009558    4550 patch_prober.go:29] interesting pod/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 01:09:46.009729 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:46.009611    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 01:09:47.007799 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:47.007762    4550 patch_prober.go:29] interesting pod/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 01:09:47.008212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:47.007835    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 01:09:47.198358 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:47.198302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:47.750837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:47.750792    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:09:48.007883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:48.007783    4550 patch_prober.go:29] interesting pod/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 01:09:48.007883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:48.007864    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 01:09:48.903631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:48.903594    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:49.020869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:49.020830    4550 patch_prober.go:29] interesting pod/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 01:09:49.021286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:49.020906    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 01:09:49.033668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:49.033640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:49.254253 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:49.254153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:49.374454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:49.373378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:49.696511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:49.696471    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:50.008130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:50.008042    4550 patch_prober.go:29] interesting pod/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 01:09:50.008130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:50.008094    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 01:09:50.747287 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:50.747248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:51.007905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:51.007821    4550 patch_prober.go:29] interesting pod/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused" start-of-body=
Nov 03 01:09:51.007905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:51.007876    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.26:15021/healthz/ready\": dial tcp 10.133.1.26:15021: connect: connection refused"
Nov 03 01:09:52.012751 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:52.012720    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:09:52.014076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:52.014053    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:09:52.403402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:52.403373    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:54.108951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:54.108909    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:54.725242 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:09:54.725206    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:09:55.726237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:55.726194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:55.834928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:55.834851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:56.201756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:56.201710    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:57.449733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:57.449691    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:58.633160 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:58.633116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:58.947331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:58.947296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:09:59.676702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:09:59.676665    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:01.411055 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:01.411020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:02.127248 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:02.127197    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"]
Nov 03 01:10:02.164251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:02.164210    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"]
Nov 03 01:10:02.253279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:02.252275    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:02.357596 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:02.357546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:03.119852 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:03.119811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:03.236766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:03.236722    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:03.358068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:03.358035    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:03.528263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:03.528209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:04.298826 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:04.298787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:05.140094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:05.140056    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:06.895668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:06.895619    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:07.128413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:07.128342    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerName="payload-processing" containerID="cri-o://5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6" gracePeriod=25
Nov 03 01:10:07.165158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:07.165103    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" podUID="67072711-766c-4236-9557-8befe2558416" containerName="payload-pre-processing" containerID="cri-o://8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604" gracePeriod=25
Nov 03 01:10:08.580664 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:08.580627    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:09.042189 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:09.042155    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:10.177246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:10.177207    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:10.263384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:10.263342    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"]
Nov 03 01:10:10.298241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:10.298202    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"]
Nov 03 01:10:10.348207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:10.348152    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:10.451076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:10.451041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:10.964203 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:10.964165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:11.513096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:11.513053    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:12.725489 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:12.725426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:14.363324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:14.363287    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:14.445900 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:14.445859    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" podUID="67072711-766c-4236-9557-8befe2558416" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:14.447324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:14.447282    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:15.263804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:15.263765    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerName="payload-processing" containerID="cri-o://47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8" gracePeriod=25
Nov 03 01:10:15.299396 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:15.299337    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerName="payload-pre-processing" containerID="cri-o://b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93" gracePeriod=25
Nov 03 01:10:15.398451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:15.398388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:16.050315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:16.050276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:17.382362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:17.382318    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:17.585217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:17.585173    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:17.800510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:17.800474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:18.154896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:18.154850    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:18.541887 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:18.541800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:18.550213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:18.550168    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"]
Nov 03 01:10:18.550497 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:18.550466    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" containerID="cri-o://b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a" gracePeriod=30
Nov 03 01:10:18.709456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:18.694840    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:18.808405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:18.807115    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:19.339383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:19.339349    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:19.753317 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:19.753278    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:20.375250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:20.375202    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:10:20.375566 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:20.375287    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:10:21.438180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:21.438144    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:22.725901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:22.725865    4550 scope.go:122] "RemoveContainer" containerID="fe44533c5a9389f2bc4c328930ca4d0a9cbd164d00f32b74ab44b06c9b6b6185"
Nov 03 01:10:22.736301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:22.736278    4550 scope.go:122] "RemoveContainer" containerID="7d85ca07d1154cdd293f8269c3a4437943b463e31cff6383d2ce7c2465b2b317"
Nov 03 01:10:22.746110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:22.746087    4550 scope.go:122] "RemoveContainer" containerID="b530735616e0e10cd83e8204190e723d4e120d258bf2a2bbe48a9ed91c89aa04"
Nov 03 01:10:22.951548 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:22.951511    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:23.651956 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.651913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:23.722482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.722093    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:23.722482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.722201    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:23.910643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.910614    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 01:10:23.941867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.941826    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.941974    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942005    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942057    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942084    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942111    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:23.942365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942187    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942238    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942278    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942319    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token\") pod \"7dff03b7-1b77-43f5-9204-29b75194a7d9\" (UID: \"7dff03b7-1b77-43f5-9204-29b75194a7d9\") "
Nov 03 01:10:23.942733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942382    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:23.942733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942570    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:23.942733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942653    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:23.942733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.942714    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:23.943108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.943040    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.943108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.943067    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.943108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.943082    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dff03b7-1b77-43f5-9204-29b75194a7d9-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.943108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.943097    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.943108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.943110    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.944695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.944668    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "kube-api-access-d58zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:23.944842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.944825    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:23.944958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.944937    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:23.944958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:23.944950    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo" pod "7dff03b7-1b77-43f5-9204-29b75194a7d9" (UID: "7dff03b7-1b77-43f5-9204-29b75194a7d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:10:24.045053 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.045009    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:24.045241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.045065    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58zj\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-kube-api-access-d58zj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:24.045241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.045076    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:24.045241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.045084    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dff03b7-1b77-43f5-9204-29b75194a7d9-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:24.136037 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.135947    4550 generic.go:356] "Generic (PLEG): container finished" podID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerID="b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a" exitCode=0
Nov 03 01:10:24.136037 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.136013    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"
Nov 03 01:10:24.136260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.136033    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" event={"ID":"7dff03b7-1b77-43f5-9204-29b75194a7d9","Type":"ContainerDied","Data":"b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a"}
Nov 03 01:10:24.136260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.136116    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2" event={"ID":"7dff03b7-1b77-43f5-9204-29b75194a7d9","Type":"ContainerDied","Data":"e072fa296edab86b47ea55e1a8395534babbef4296b785640f2a1fce98cf4712"}
Nov 03 01:10:24.136260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.136140    4550 scope.go:122] "RemoveContainer" containerID="b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a"
Nov 03 01:10:24.147660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.147632    4550 scope.go:122] "RemoveContainer" containerID="0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"
Nov 03 01:10:24.158244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.158217    4550 scope.go:122] "RemoveContainer" containerID="b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a"
Nov 03 01:10:24.158664 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:24.158630    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a\": container with ID starting with b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a not found: ID does not exist" containerID="b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a"
Nov 03 01:10:24.158806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.158677    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a"} err="failed to get container status \"b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a\": rpc error: code = NotFound desc = could not find container \"b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a\": container with ID starting with b8018b432a2e80abeecd351ec658611bbb11ff808013b1e08cfa57d537cb791a not found: ID does not exist"
Nov 03 01:10:24.158806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.158704    4550 scope.go:122] "RemoveContainer" containerID="0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"
Nov 03 01:10:24.159038 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:24.159009    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821\": container with ID starting with 0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821 not found: ID does not exist" containerID="0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"
Nov 03 01:10:24.159102 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.159048    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821"} err="failed to get container status \"0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821\": rpc error: code = NotFound desc = could not find container \"0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821\": container with ID starting with 0184ae8c0ffbd404f03cdde0dd960f85ec1fc64a25d1f3ff2fb5a3bcc453e821 not found: ID does not exist"
Nov 03 01:10:24.191495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.191457    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"]
Nov 03 01:10:24.199804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.199756    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d4d8c2-openshift-default-c64d656b5-l4dk2"]
Nov 03 01:10:24.431069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.431029    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" podUID="67072711-766c-4236-9557-8befe2558416" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:24.431069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:24.431067    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:24.724906 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:24.724823    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:10:25.429854 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:25.429813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:25.808842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:25.808759    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" path="/var/lib/kubelet/pods/7dff03b7-1b77-43f5-9204-29b75194a7d9/volumes"
Nov 03 01:10:26.236934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:26.236881    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:26.356390 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:26.356351    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:26.708921 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:26.708885    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:27.352730 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.352648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:27.491159 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.491123    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 01:10:27.558358 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.558319    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 01:10:27.615152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.615099    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng\" (UniqueName: \"kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng\") pod \"99f13630-6b7e-4799-bc76-20e7d84fef3b\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") "
Nov 03 01:10:27.615360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.615330    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume\") pod \"99f13630-6b7e-4799-bc76-20e7d84fef3b\" (UID: \"99f13630-6b7e-4799-bc76-20e7d84fef3b\") "
Nov 03 01:10:27.615935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.615908    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume" pod "99f13630-6b7e-4799-bc76-20e7d84fef3b" (UID: "99f13630-6b7e-4799-bc76-20e7d84fef3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:27.617936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.617912    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng" pod "99f13630-6b7e-4799-bc76-20e7d84fef3b" (UID: "99f13630-6b7e-4799-bc76-20e7d84fef3b"). InnerVolumeSpecName "kube-api-access-8lxng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:27.717069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.716970    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume\") pod \"67072711-766c-4236-9557-8befe2558416\" (UID: \"67072711-766c-4236-9557-8befe2558416\") "
Nov 03 01:10:27.717607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.717199    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk\" (UniqueName: \"kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk\") pod \"67072711-766c-4236-9557-8befe2558416\" (UID: \"67072711-766c-4236-9557-8befe2558416\") "
Nov 03 01:10:27.717607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.717344    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume" pod "67072711-766c-4236-9557-8befe2558416" (UID: "67072711-766c-4236-9557-8befe2558416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:27.717810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.717788    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13630-6b7e-4799-bc76-20e7d84fef3b-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:27.717874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.717815    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lxng\" (UniqueName: \"kubernetes.io/projected/99f13630-6b7e-4799-bc76-20e7d84fef3b-kube-api-access-8lxng\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:27.717874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.717831    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67072711-766c-4236-9557-8befe2558416-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:27.719387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.719366    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk" pod "67072711-766c-4236-9557-8befe2558416" (UID: "67072711-766c-4236-9557-8befe2558416"). InnerVolumeSpecName "kube-api-access-5qvxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:27.819134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:27.819097    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qvxk\" (UniqueName: \"kubernetes.io/projected/67072711-766c-4236-9557-8befe2558416-kube-api-access-5qvxk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:28.168972 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.168930    4550 generic.go:356] "Generic (PLEG): container finished" podID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerID="5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6" exitCode=0
Nov 03 01:10:28.169192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.168977    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"
Nov 03 01:10:28.169192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.169008    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" event={"ID":"99f13630-6b7e-4799-bc76-20e7d84fef3b","Type":"ContainerDied","Data":"5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6"}
Nov 03 01:10:28.169192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.169062    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc" event={"ID":"99f13630-6b7e-4799-bc76-20e7d84fef3b","Type":"ContainerDied","Data":"e11f662161bc7d2b53e57698fe04b00171967ae8cf70534aaf49db68fbccfc90"}
Nov 03 01:10:28.169192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.169077    4550 scope.go:122] "RemoveContainer" containerID="5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6"
Nov 03 01:10:28.172003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.170908    4550 generic.go:356] "Generic (PLEG): container finished" podID="67072711-766c-4236-9557-8befe2558416" containerID="8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604" exitCode=0
Nov 03 01:10:28.172003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.170961    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"
Nov 03 01:10:28.172003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.170989    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" event={"ID":"67072711-766c-4236-9557-8befe2558416","Type":"ContainerDied","Data":"8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604"}
Nov 03 01:10:28.172003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.171051    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d" event={"ID":"67072711-766c-4236-9557-8befe2558416","Type":"ContainerDied","Data":"ada0e84a8df0c951697eacfebadcaac58975810e13caec570ac4e9fc5679aa5f"}
Nov 03 01:10:28.182252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.182227    4550 scope.go:122] "RemoveContainer" containerID="5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6"
Nov 03 01:10:28.182710 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:28.182687    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6\": container with ID starting with 5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6 not found: ID does not exist" containerID="5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6"
Nov 03 01:10:28.182780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.182721    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6"} err="failed to get container status \"5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6\": rpc error: code = NotFound desc = could not find container \"5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6\": container with ID starting with 5d9081c768c1e5dce77843bd986787ece59bafbdac5d43d04385352bb7effdc6 not found: ID does not exist"
Nov 03 01:10:28.182780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.182741    4550 scope.go:122] "RemoveContainer" containerID="8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604"
Nov 03 01:10:28.194520 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.194476    4550 scope.go:122] "RemoveContainer" containerID="8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604"
Nov 03 01:10:28.194903 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:28.194879    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604\": container with ID starting with 8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604 not found: ID does not exist" containerID="8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604"
Nov 03 01:10:28.195008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.194912    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604"} err="failed to get container status \"8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604\": rpc error: code = NotFound desc = could not find container \"8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604\": container with ID starting with 8a99403cc8a7b5ceea25fc7b9580ddd1422a1c16632340874a0076cb397d2604 not found: ID does not exist"
Nov 03 01:10:28.224778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.224737    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"]
Nov 03 01:10:28.237795 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.237761    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-955dc3-567769895c-s4dwc"]
Nov 03 01:10:28.262197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.262155    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"]
Nov 03 01:10:28.265985 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:28.265950    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-955dc3-7ddf4c859-wdc9d"]
Nov 03 01:10:29.133347 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:29.133300    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:29.809936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:29.809849    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67072711-766c-4236-9557-8befe2558416" path="/var/lib/kubelet/pods/67072711-766c-4236-9557-8befe2558416/volumes"
Nov 03 01:10:29.810343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:29.810321    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" path="/var/lib/kubelet/pods/99f13630-6b7e-4799-bc76-20e7d84fef3b/volumes"
Nov 03 01:10:30.786052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:30.786012    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:32.540804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:32.540772    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:32.726999 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:32.726958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:33.720026 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:33.719985    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:33.720026 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:33.719985    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:33.919323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:33.919287    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:34.017773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:34.017673    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:34.851178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:34.851136    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:34.953712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:34.953674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:35.192174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.192132    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"]
Nov 03 01:10:35.192473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.192420    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" containerID="cri-o://0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a" gracePeriod=30
Nov 03 01:10:35.230871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.230832    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:35.622875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.622845    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 01:10:35.660788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.660758    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 01:10:35.761171 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.761085    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume\") pod \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") "
Nov 03 01:10:35.761357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.761181    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume\") pod \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") "
Nov 03 01:10:35.761357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.761224    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw\" (UniqueName: \"kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw\") pod \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\" (UID: \"4f2cf1ee-59d3-457c-b737-b672bc49eb58\") "
Nov 03 01:10:35.761515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.761390    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr\" (UniqueName: \"kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr\") pod \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\" (UID: \"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7\") "
Nov 03 01:10:35.761663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.761637    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume" pod "4f2cf1ee-59d3-457c-b737-b672bc49eb58" (UID: "4f2cf1ee-59d3-457c-b737-b672bc49eb58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:35.761745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.761723    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume" pod "fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" (UID: "fbd0c6ce-8286-40af-b06f-bbcfd8f749e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:35.762211 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.762188    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2cf1ee-59d3-457c-b737-b672bc49eb58-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:35.762295 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.762217    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:35.763632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.763609    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw" pod "4f2cf1ee-59d3-457c-b737-b672bc49eb58" (UID: "4f2cf1ee-59d3-457c-b737-b672bc49eb58"). InnerVolumeSpecName "kube-api-access-nrjhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:35.763722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.763703    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr" pod "fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" (UID: "fbd0c6ce-8286-40af-b06f-bbcfd8f749e7"). InnerVolumeSpecName "kube-api-access-6f8pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:35.863426 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.863387    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjhw\" (UniqueName: \"kubernetes.io/projected/4f2cf1ee-59d3-457c-b737-b672bc49eb58-kube-api-access-nrjhw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:35.863426 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:35.863422    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f8pr\" (UniqueName: \"kubernetes.io/projected/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7-kube-api-access-6f8pr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:36.013496 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.013377    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:36.265156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.265049    4550 generic.go:356] "Generic (PLEG): container finished" podID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerID="47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8" exitCode=0
Nov 03 01:10:36.265156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.265112    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"
Nov 03 01:10:36.265705 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.265139    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" event={"ID":"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7","Type":"ContainerDied","Data":"47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8"}
Nov 03 01:10:36.265705 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.265209    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx" event={"ID":"fbd0c6ce-8286-40af-b06f-bbcfd8f749e7","Type":"ContainerDied","Data":"16e7f1eaca27eee1897577e819df1d5258fec207e87841e2707cd227d97ab8d8"}
Nov 03 01:10:36.265705 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.265241    4550 scope.go:122] "RemoveContainer" containerID="47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8"
Nov 03 01:10:36.267027 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.266884    4550 generic.go:356] "Generic (PLEG): container finished" podID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerID="b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93" exitCode=0
Nov 03 01:10:36.267103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.267047    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" event={"ID":"4f2cf1ee-59d3-457c-b737-b672bc49eb58","Type":"ContainerDied","Data":"b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93"}
Nov 03 01:10:36.267103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.267086    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf" event={"ID":"4f2cf1ee-59d3-457c-b737-b672bc49eb58","Type":"ContainerDied","Data":"70c80e44aeecb325a5d798826e486e5d0ab85046e402d5d8cd3d966adc375d94"}
Nov 03 01:10:36.267187 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.267144    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"
Nov 03 01:10:36.278412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.278394    4550 scope.go:122] "RemoveContainer" containerID="47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8"
Nov 03 01:10:36.278724 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:36.278697    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8\": container with ID starting with 47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8 not found: ID does not exist" containerID="47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8"
Nov 03 01:10:36.278833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.278734    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8"} err="failed to get container status \"47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8\": rpc error: code = NotFound desc = could not find container \"47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8\": container with ID starting with 47480f0d2b67cf48e1c1cd98555bb8fe8739772d41cb1dfdd6ad2191306330b8 not found: ID does not exist"
Nov 03 01:10:36.278833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.278752    4550 scope.go:122] "RemoveContainer" containerID="b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93"
Nov 03 01:10:36.292831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.292804    4550 scope.go:122] "RemoveContainer" containerID="b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93"
Nov 03 01:10:36.293191 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:36.293164    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93\": container with ID starting with b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93 not found: ID does not exist" containerID="b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93"
Nov 03 01:10:36.293255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.293203    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93"} err="failed to get container status \"b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93\": rpc error: code = NotFound desc = could not find container \"b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93\": container with ID starting with b7709b2fde96bcf14ffe7962a968e45f053715048cd54c3ffffcff71050b6e93 not found: ID does not exist"
Nov 03 01:10:36.319901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.319867    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"]
Nov 03 01:10:36.331934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.331902    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d4d8c2-7584b574f9-vtmnf"]
Nov 03 01:10:36.360891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.360860    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"]
Nov 03 01:10:36.366186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.366156    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d4d8c2-5c9965d747-pdjmx"]
Nov 03 01:10:36.609260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:36.609157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:37.315141 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:37.315102    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:10:37.315579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:37.315166    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:10:37.809221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:37.809183    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" path="/var/lib/kubelet/pods/4f2cf1ee-59d3-457c-b737-b672bc49eb58/volumes"
Nov 03 01:10:37.809715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:37.809699    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" path="/var/lib/kubelet/pods/fbd0c6ce-8286-40af-b06f-bbcfd8f749e7/volumes"
Nov 03 01:10:38.321074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:38.321043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:39.360866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:39.360836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:39.506773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:39.506735    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:40.200132 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.200089    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:40.307797 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.307611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:40.496366 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.496338    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 01:10:40.531411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531380    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.531411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531420    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.531785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531565    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.531785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531605    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.531785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531673    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.531785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531728    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.531785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531759    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.532075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531812    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.532075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.531899    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token\") pod \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\" (UID: \"805b264e-a5ef-4cbc-af02-a7300e4ccaf9\") "
Nov 03 01:10:40.532986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.532901    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:40.533314 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.533263    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:40.533474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.533390    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:40.533474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.533455    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:40.533853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.533830    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:40.534228 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.534209    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.534308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.534232    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.534308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.534246    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.534308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.534258    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.534308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.534272    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.560818 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.560778    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:10:40.562821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.562744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:40.572692 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.572489    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:40.577479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.574174    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:40.586008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.582417    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"]
Nov 03 01:10:40.586008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.583180    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy" containerID="cri-o://62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71" gracePeriod=30
Nov 03 01:10:40.595498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.595467    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb" pod "805b264e-a5ef-4cbc-af02-a7300e4ccaf9" (UID: "805b264e-a5ef-4cbc-af02-a7300e4ccaf9"). InnerVolumeSpecName "kube-api-access-d5cmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:40.635415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.635366    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.635415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.635413    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5cmb\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-kube-api-access-d5cmb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.635415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.635442    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:40.635415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:40.635456    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805b264e-a5ef-4cbc-af02-a7300e4ccaf9-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:41.323534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.323499    4550 generic.go:356] "Generic (PLEG): container finished" podID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerID="0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a" exitCode=0
Nov 03 01:10:41.323776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.323551    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"
Nov 03 01:10:41.323776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.323582    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" event={"ID":"805b264e-a5ef-4cbc-af02-a7300e4ccaf9","Type":"ContainerDied","Data":"0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a"}
Nov 03 01:10:41.323776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.323636    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg" event={"ID":"805b264e-a5ef-4cbc-af02-a7300e4ccaf9","Type":"ContainerDied","Data":"c2fd26e4a3085683e1e641722eb5da032b26e849b4a72f61310fec30909fc4a4"}
Nov 03 01:10:41.323776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.323659    4550 scope.go:122] "RemoveContainer" containerID="0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a"
Nov 03 01:10:41.335037 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.334881    4550 scope.go:122] "RemoveContainer" containerID="988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"
Nov 03 01:10:41.346000 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.345971    4550 scope.go:122] "RemoveContainer" containerID="0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a"
Nov 03 01:10:41.346344 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:41.346310    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a\": container with ID starting with 0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a not found: ID does not exist" containerID="0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a"
Nov 03 01:10:41.346415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.346352    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a"} err="failed to get container status \"0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a\": rpc error: code = NotFound desc = could not find container \"0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a\": container with ID starting with 0076917043a02f3dcb61574ee82a1a1bd607c0e2c60bff2c02fb9f85e481237a not found: ID does not exist"
Nov 03 01:10:41.346415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.346372    4550 scope.go:122] "RemoveContainer" containerID="988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"
Nov 03 01:10:41.346774 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:41.346756    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b\": container with ID starting with 988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b not found: ID does not exist" containerID="988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"
Nov 03 01:10:41.346843 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.346780    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b"} err="failed to get container status \"988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b\": rpc error: code = NotFound desc = could not find container \"988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b\": container with ID starting with 988af5c0ea29ab763ec257a08af2bb10e67b75c6ca8f171d47ebee296a8e402b not found: ID does not exist"
Nov 03 01:10:41.379164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.378830    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:41.414944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.414905    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"]
Nov 03 01:10:41.429462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.425972    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-955dc3-openshift-default-76559d4f5c-4qqrg"]
Nov 03 01:10:41.493156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.493049    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:41.809181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:41.809100    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" path="/var/lib/kubelet/pods/805b264e-a5ef-4cbc-af02-a7300e4ccaf9/volumes"
Nov 03 01:10:42.246451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:42.246391    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:42.691956 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:42.691918    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:44.024111 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:44.024074    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:45.740640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:45.740593    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:45.889485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:45.889455    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 01:10:46.038496 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038423    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038572    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038636    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038684    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038725    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:46.038906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038747    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038770    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038803    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038854    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.038906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.038886    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x\") pod \"9641d0eb-647f-4294-a396-647398bfaab3\" (UID: \"9641d0eb-647f-4294-a396-647398bfaab3\") "
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039033    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039094    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039242    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039272    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039544    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039562    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9641d0eb-647f-4294-a396-647398bfaab3-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039571    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039579    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.039699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.039590    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.040921 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.040901    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:10:46.041206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.041191    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:46.041374 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.041361    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:46.041473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.041416    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x" pod "9641d0eb-647f-4294-a396-647398bfaab3" (UID: "9641d0eb-647f-4294-a396-647398bfaab3"). InnerVolumeSpecName "kube-api-access-qw87x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:46.141340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.141254    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.141340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.141286    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9641d0eb-647f-4294-a396-647398bfaab3-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.141340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.141295    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9641d0eb-647f-4294-a396-647398bfaab3-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.141340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.141305    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw87x\" (UniqueName: \"kubernetes.io/projected/9641d0eb-647f-4294-a396-647398bfaab3-kube-api-access-qw87x\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:10:46.364510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.364469    4550 generic.go:356] "Generic (PLEG): container finished" podID="9641d0eb-647f-4294-a396-647398bfaab3" containerID="62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71" exitCode=0
Nov 03 01:10:46.364736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.364527    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"
Nov 03 01:10:46.364736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.364542    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" event={"ID":"9641d0eb-647f-4294-a396-647398bfaab3","Type":"ContainerDied","Data":"62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71"}
Nov 03 01:10:46.364736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.364586    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn" event={"ID":"9641d0eb-647f-4294-a396-647398bfaab3","Type":"ContainerDied","Data":"517e83b8294da933c018596ee9f8c76313caadd1048ee4dce25319759a0fe5d3"}
Nov 03 01:10:46.364736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.364609    4550 scope.go:122] "RemoveContainer" containerID="62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71"
Nov 03 01:10:46.376872 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.376852    4550 scope.go:122] "RemoveContainer" containerID="62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71"
Nov 03 01:10:46.377205 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:46.377186    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71\": container with ID starting with 62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71 not found: ID does not exist" containerID="62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71"
Nov 03 01:10:46.377278 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.377220    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71"} err="failed to get container status \"62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71\": rpc error: code = NotFound desc = could not find container \"62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71\": container with ID starting with 62fb086a14ebb17cf550d1ac126dc16233450802f71e0a26da6468687df82e71 not found: ID does not exist"
Nov 03 01:10:46.421977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.421939    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"]
Nov 03 01:10:46.434715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:46.434677    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-rjmvn"]
Nov 03 01:10:47.561752 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:47.561715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:47.775634 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:47.775599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:47.836812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:47.836689    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9641d0eb-647f-4294-a396-647398bfaab3" path="/var/lib/kubelet/pods/9641d0eb-647f-4294-a396-647398bfaab3/volumes"
Nov 03 01:10:47.925697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:47.925661    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:48.101522 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:48.101413    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:49.145962 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:49.145927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:49.338794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:49.338759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:50.939419 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:50.939388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:52.643452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:52.643393    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:54.410108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:54.410064    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:54.725399 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:10:54.725305    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:10:55.465144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:55.465105    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:55.996337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:55.996296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:57.637510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:57.637472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:59.254705 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.254663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.676864    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"]
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678143    4550 cpu_manager.go:414] "containerMap: removing container" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerName="payload-pre-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678169    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerName="payload-pre-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678201    4550 cpu_manager.go:414] "containerMap: removing container" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678209    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678247    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678255    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678309    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678317    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678338    4550 cpu_manager.go:414] "containerMap: removing container" podUID="67072711-766c-4236-9557-8befe2558416" containerName="payload-pre-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678345    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67072711-766c-4236-9557-8befe2558416" containerName="payload-pre-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678370    4550 cpu_manager.go:414] "containerMap: removing container" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678378    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678402    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerName="payload-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678408    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerName="payload-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678444    4550 cpu_manager.go:414] "containerMap: removing container" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678451    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa828ae-d375-4578-9485-71ed468a41e1" containerName="istio-proxy"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678471    4550 cpu_manager.go:414] "containerMap: removing container" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerName="payload-processing"
Nov 03 01:10:59.683452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.678479    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerName="payload-processing"
Nov 03 01:10:59.684761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684733    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2cf1ee-59d3-457c-b737-b672bc49eb58" containerName="payload-pre-processing"
Nov 03 01:10:59.684838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684769    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy"
Nov 03 01:10:59.684838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684789    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy"
Nov 03 01:10:59.684838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684812    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9641d0eb-647f-4294-a396-647398bfaab3" containerName="istio-proxy"
Nov 03 01:10:59.684838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684822    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="67072711-766c-4236-9557-8befe2558416" containerName="payload-pre-processing"
Nov 03 01:10:59.684838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684830    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dff03b7-1b77-43f5-9204-29b75194a7d9" containerName="istio-proxy"
Nov 03 01:10:59.684838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684839    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd0c6ce-8286-40af-b06f-bbcfd8f749e7" containerName="payload-processing"
Nov 03 01:10:59.685143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684855    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f13630-6b7e-4799-bc76-20e7d84fef3b" containerName="payload-processing"
Nov 03 01:10:59.685143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.684872    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy"
Nov 03 01:10:59.691884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.691854    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.698529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.696988    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"]
Nov 03 01:10:59.710639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.709868    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:10:59.724353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.724308    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:10:59.854357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.854309    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.854560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.854490    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88bt\" (UniqueName: \"kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.854560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.854543    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.955907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.955822    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.956078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.955937    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88bt\" (UniqueName: \"kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.956078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.955988    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.956653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.956621    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.959077 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.959041    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:10:59.977708 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:10:59.977664    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88bt\" (UniqueName: \"kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt\") pod \"maas-api-revoke-sub-a7c4a5eb-hrn7q\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:11:00.018081 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:00.018046    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:11:00.322596 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:11:00.322552    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d68abf_a64a_40d0_abfb_a04df36a425c.slice/crio-ba17de31a910b471abc881ffc556b480a08b786a99659655443f2f13b4d81fd3 WatchSource:0}: Error finding container ba17de31a910b471abc881ffc556b480a08b786a99659655443f2f13b4d81fd3: Status 404 returned error can't find the container with id ba17de31a910b471abc881ffc556b480a08b786a99659655443f2f13b4d81fd3
Nov 03 01:11:00.325171 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:00.325142    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"]
Nov 03 01:11:00.537067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:00.537025    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q" event={"ID":"05d68abf-a64a-40d0-abfb-a04df36a425c","Type":"ContainerStarted","Data":"ba17de31a910b471abc881ffc556b480a08b786a99659655443f2f13b4d81fd3"}
Nov 03 01:11:01.051170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:01.051137    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:01.546089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:01.546048    4550 generic.go:356] "Generic (PLEG): container finished" podID="05d68abf-a64a-40d0-abfb-a04df36a425c" containerID="3ac5a7589801c2b195a73fd78baaf5fd91deb521ed5edfc17ee7e18db8869f0e" exitCode=0
Nov 03 01:11:01.546568 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:01.546095    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q" event={"ID":"05d68abf-a64a-40d0-abfb-a04df36a425c","Type":"ContainerDied","Data":"3ac5a7589801c2b195a73fd78baaf5fd91deb521ed5edfc17ee7e18db8869f0e"}
Nov 03 01:11:02.643376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.643334    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:02.760959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.760922    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:11:02.806072 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.805990    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca\") pod \"05d68abf-a64a-40d0-abfb-a04df36a425c\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") "
Nov 03 01:11:02.806259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.806162    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca\") pod \"05d68abf-a64a-40d0-abfb-a04df36a425c\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") "
Nov 03 01:11:02.806333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.806268    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt\" (UniqueName: \"kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt\") pod \"05d68abf-a64a-40d0-abfb-a04df36a425c\" (UID: \"05d68abf-a64a-40d0-abfb-a04df36a425c\") "
Nov 03 01:11:02.806396 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.806348    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca" pod "05d68abf-a64a-40d0-abfb-a04df36a425c" (UID: "05d68abf-a64a-40d0-abfb-a04df36a425c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:02.806967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.806907    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:02.808505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.808478    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca" pod "05d68abf-a64a-40d0-abfb-a04df36a425c" (UID: "05d68abf-a64a-40d0-abfb-a04df36a425c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:02.808827 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.808803    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt" pod "05d68abf-a64a-40d0-abfb-a04df36a425c" (UID: "05d68abf-a64a-40d0-abfb-a04df36a425c"). InnerVolumeSpecName "kube-api-access-v88bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:02.908451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.908400    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05d68abf-a64a-40d0-abfb-a04df36a425c-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:02.908649 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:02.908470    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v88bt\" (UniqueName: \"kubernetes.io/projected/05d68abf-a64a-40d0-abfb-a04df36a425c-kube-api-access-v88bt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:03.570297 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:03.570265    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"
Nov 03 01:11:03.570563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:03.570265    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q" event={"ID":"05d68abf-a64a-40d0-abfb-a04df36a425c","Type":"ContainerDied","Data":"ba17de31a910b471abc881ffc556b480a08b786a99659655443f2f13b4d81fd3"}
Nov 03 01:11:03.570563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:03.570407    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba17de31a910b471abc881ffc556b480a08b786a99659655443f2f13b4d81fd3"
Nov 03 01:11:04.307963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:04.307926    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:05.882764 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:05.882721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:07.637460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:07.637404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:09.358498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:09.358448    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:10.058660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.058623    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"]
Nov 03 01:11:10.059212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.059196    4550 cpu_manager.go:414] "containerMap: removing container" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy"
Nov 03 01:11:10.059259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.059214    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805b264e-a5ef-4cbc-af02-a7300e4ccaf9" containerName="istio-proxy"
Nov 03 01:11:10.059259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.059237    4550 cpu_manager.go:414] "containerMap: removing container" podUID="05d68abf-a64a-40d0-abfb-a04df36a425c" containerName="revoke-keys"
Nov 03 01:11:10.059259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.059242    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d68abf-a64a-40d0-abfb-a04df36a425c" containerName="revoke-keys"
Nov 03 01:11:10.059364 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.059319    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d68abf-a64a-40d0-abfb-a04df36a425c" containerName="revoke-keys"
Nov 03 01:11:10.062925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.062908    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.080801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.080772    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"]
Nov 03 01:11:10.093475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.090121    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:11:10.093475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.090168    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:11:10.213086 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.213046    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.213086 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.213091    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.213321 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.213106    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4pvg\" (UniqueName: \"kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.315060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.314953    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.315060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.315006    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.315334 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.315136    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4pvg\" (UniqueName: \"kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.315684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.315664    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.317456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.317407    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.335161 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.335134    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4pvg\" (UniqueName: \"kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg\") pod \"maas-api-revoke-sub-ecd14b15-n6qlm\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.377328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.377290    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:10.682672 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:11:10.682635    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6522ca_5488_47da_ba9a_3009bfed6182.slice/crio-7adaff61155aa2095bc03d449c289f8831aae9cdf9f125ea10130319d46e79d9 WatchSource:0}: Error finding container 7adaff61155aa2095bc03d449c289f8831aae9cdf9f125ea10130319d46e79d9: Status 404 returned error can't find the container with id 7adaff61155aa2095bc03d449c289f8831aae9cdf9f125ea10130319d46e79d9
Nov 03 01:11:10.683998 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:10.683950    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"]
Nov 03 01:11:11.005139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:11.005099    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:11.636106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:11.636071    4550 generic.go:356] "Generic (PLEG): container finished" podID="5e6522ca-5488-47da-ba9a-3009bfed6182" containerID="265620a0affcbf536fa8d2ffd2a352347e67c9178bd34a6e7a8c650bf2c532d5" exitCode=0
Nov 03 01:11:11.636566 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:11.636154    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm" event={"ID":"5e6522ca-5488-47da-ba9a-3009bfed6182","Type":"ContainerDied","Data":"265620a0affcbf536fa8d2ffd2a352347e67c9178bd34a6e7a8c650bf2c532d5"}
Nov 03 01:11:11.636566 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:11.636207    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm" event={"ID":"5e6522ca-5488-47da-ba9a-3009bfed6182","Type":"ContainerStarted","Data":"7adaff61155aa2095bc03d449c289f8831aae9cdf9f125ea10130319d46e79d9"}
Nov 03 01:11:12.637479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.637442    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:12.821079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.821047    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:12.951991 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.951952    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca\") pod \"5e6522ca-5488-47da-ba9a-3009bfed6182\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") "
Nov 03 01:11:12.952191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.952024    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca\") pod \"5e6522ca-5488-47da-ba9a-3009bfed6182\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") "
Nov 03 01:11:12.952191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.952099    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg\" (UniqueName: \"kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg\") pod \"5e6522ca-5488-47da-ba9a-3009bfed6182\" (UID: \"5e6522ca-5488-47da-ba9a-3009bfed6182\") "
Nov 03 01:11:12.952286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.952267    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca" pod "5e6522ca-5488-47da-ba9a-3009bfed6182" (UID: "5e6522ca-5488-47da-ba9a-3009bfed6182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:12.952873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.952850    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:12.954476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.954408    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg" pod "5e6522ca-5488-47da-ba9a-3009bfed6182" (UID: "5e6522ca-5488-47da-ba9a-3009bfed6182"). InnerVolumeSpecName "kube-api-access-r4pvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:12.954611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:12.954593    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca" pod "5e6522ca-5488-47da-ba9a-3009bfed6182" (UID: "5e6522ca-5488-47da-ba9a-3009bfed6182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:13.054125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:13.054084    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6522ca-5488-47da-ba9a-3009bfed6182-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:13.054125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:13.054118    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4pvg\" (UniqueName: \"kubernetes.io/projected/5e6522ca-5488-47da-ba9a-3009bfed6182-kube-api-access-r4pvg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:13.627970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:13.627929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:13.649843 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:13.649810    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"
Nov 03 01:11:13.650357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:13.649842    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm" event={"ID":"5e6522ca-5488-47da-ba9a-3009bfed6182","Type":"ContainerDied","Data":"7adaff61155aa2095bc03d449c289f8831aae9cdf9f125ea10130319d46e79d9"}
Nov 03 01:11:13.650357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:13.649925    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7adaff61155aa2095bc03d449c289f8831aae9cdf9f125ea10130319d46e79d9"
Nov 03 01:11:14.125588 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:14.125559    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:14.839284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:14.839241    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:15.517313 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:15.517279    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:16.520821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:16.520787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:18.135482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:18.135421    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:18.726003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:18.725968    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:18.831489 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:18.831170    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:19.779636 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:19.779596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:19.961710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:19.961674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:21.008214 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:21.008181    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:21.725370 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:21.725334    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:22.788905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:22.788869    4550 scope.go:122] "RemoveContainer" containerID="4142e6f84257b8948a3ff1305ead47ee01dc7d93e1937cbf50204e8603322138"
Nov 03 01:11:23.337504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:23.337464    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:23.441586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:23.441554    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:24.116657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:24.116621    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:24.474805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:24.474768    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"]
Nov 03 01:11:24.475332 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:24.475296    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" containerID="cri-o://3b39f6071d80aa5abdc3b8aa28c54f85b700993b268ec29828c115fd003c455b" gracePeriod=30
Nov 03 01:11:24.511382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:24.510339    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:24.744906 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:11:24.744815    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:11:25.025461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:25.025327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:25.275711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:25.274966    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:25.845096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:25.845061    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:25.945869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:25.945837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:27.531934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:27.531902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:28.609835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:28.609795    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:28.710087 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:28.710050    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:28.913687 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:28.913644    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:11:28.913917 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:28.913729    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:11:29.494016 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.493978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:29.590460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.590410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:29.812809 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.812725    4550 generic.go:356] "Generic (PLEG): container finished" podID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerID="3b39f6071d80aa5abdc3b8aa28c54f85b700993b268ec29828c115fd003c455b" exitCode=0
Nov 03 01:11:29.812809 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.812791    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" event={"ID":"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37","Type":"ContainerDied","Data":"3b39f6071d80aa5abdc3b8aa28c54f85b700993b268ec29828c115fd003c455b"}
Nov 03 01:11:29.813276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.812852    4550 scope.go:122] "RemoveContainer" containerID="275b9e73974cc46ff6e8e5972ae42fe7ee099bb7cac4551784e2a95f78073222"
Nov 03 01:11:29.932863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.931973    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"]
Nov 03 01:11:29.932863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.932351    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy" containerID="cri-o://d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3" gracePeriod=30
Nov 03 01:11:29.955981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:29.955934    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:30.293617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.293589    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 01:11:30.415737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.415699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:30.458088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.458046    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.458088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.458115    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.459500 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.458283    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.459656 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.458387    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:30.459656 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.458535    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:30.459776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.459710    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.459776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.459748    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.459898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.459800    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.459898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.459839    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.459898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.459869    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.460075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.459907    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy\") pod \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\" (UID: \"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37\") "
Nov 03 01:11:30.460338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.460232    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:30.462319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.460725    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.462319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.460759    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.462319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.460777    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.462319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.460305    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:30.462319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.460938    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:30.462319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.462268    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:11:30.463919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.463884    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:30.464069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.464016    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:30.464310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.464286    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7" pod "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" (UID: "9a5e5569-e612-4dc2-b130-f3b8c8fb7b37"). InnerVolumeSpecName "kube-api-access-pjbt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:30.562848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.562806    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.562848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.562844    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.562848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.562857    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.563177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.562868    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.563177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.562882    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbt7\" (UniqueName: \"kubernetes.io/projected/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-kube-api-access-pjbt7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.563177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.562897    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:30.825504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.822131    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"
Nov 03 01:11:30.825504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.822202    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9" event={"ID":"9a5e5569-e612-4dc2-b130-f3b8c8fb7b37","Type":"ContainerDied","Data":"e9a89f887741966600aa75075710778d6155667355af735fb6b54e1248a34d60"}
Nov 03 01:11:30.825504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.822309    4550 scope.go:122] "RemoveContainer" containerID="3b39f6071d80aa5abdc3b8aa28c54f85b700993b268ec29828c115fd003c455b"
Nov 03 01:11:30.844748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.844721    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"]
Nov 03 01:11:30.845382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845361    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy"
Nov 03 01:11:30.845530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845384    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy"
Nov 03 01:11:30.845530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845408    4550 cpu_manager.go:414] "containerMap: removing container" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy"
Nov 03 01:11:30.845530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845414    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy"
Nov 03 01:11:30.845530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845455    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5e6522ca-5488-47da-ba9a-3009bfed6182" containerName="revoke-keys"
Nov 03 01:11:30.845530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845463    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6522ca-5488-47da-ba9a-3009bfed6182" containerName="revoke-keys"
Nov 03 01:11:30.845720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845603    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy"
Nov 03 01:11:30.845720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845617    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" containerName="istio-proxy"
Nov 03 01:11:30.845720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.845626    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6522ca-5488-47da-ba9a-3009bfed6182" containerName="revoke-keys"
Nov 03 01:11:30.850138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.850120    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:30.868557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.867463    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:11:30.868557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.867674    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:11:30.875608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.875569    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"]
Nov 03 01:11:30.886508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.886476    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"]
Nov 03 01:11:30.886661 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.886533    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-909268-openshift-default-846d5db8fc-r66k9"]
Nov 03 01:11:30.970185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.970146    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttc7k\" (UniqueName: \"kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:30.970423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.970391    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:30.970512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:30.970493    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.039496 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.036566    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"]
Nov 03 01:11:31.039946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.039917    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"]
Nov 03 01:11:31.042766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.042743    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19aafc99-kk4dj"]
Nov 03 01:11:31.045295 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.045273    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-715484be-5tk98"]
Nov 03 01:11:31.071668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.071639    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.071846 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.071686    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.071846 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.071764    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttc7k\" (UniqueName: \"kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.072514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.072483    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.074292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.074269    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.092200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.092141    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttc7k\" (UniqueName: \"kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k\") pod \"maas-api-revoke-sub-d752cb1a-bcnpm\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.169106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.169067    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:31.217610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.217116    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:31.482374 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.482344    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"]
Nov 03 01:11:31.484234 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:11:31.484200    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0371001_6166_4147_b499_dfaedc21c26a.slice/crio-e955c6e081e8e85f6dbfa535b0f4a20337d8152d3ff71b880189367cddba2a90 WatchSource:0}: Error finding container e955c6e081e8e85f6dbfa535b0f4a20337d8152d3ff71b880189367cddba2a90: Status 404 returned error can't find the container with id e955c6e081e8e85f6dbfa535b0f4a20337d8152d3ff71b880189367cddba2a90
Nov 03 01:11:31.811361 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.811330    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735f6cd4-6968-4f31-a1be-a1d704473a0d" path="/var/lib/kubelet/pods/735f6cd4-6968-4f31-a1be-a1d704473a0d/volumes"
Nov 03 01:11:31.811907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.811888    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894553c8-7dde-46d1-90f5-b68d5c772127" path="/var/lib/kubelet/pods/894553c8-7dde-46d1-90f5-b68d5c772127/volumes"
Nov 03 01:11:31.812545 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.812503    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5e5569-e612-4dc2-b130-f3b8c8fb7b37" path="/var/lib/kubelet/pods/9a5e5569-e612-4dc2-b130-f3b8c8fb7b37/volumes"
Nov 03 01:11:31.816662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.816635    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:31.844507 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.844471    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm" event={"ID":"c0371001-6166-4147-b499-dfaedc21c26a","Type":"ContainerStarted","Data":"ddf55d0803314d0fcaafa46b22f197d653692bc99d140761786b661734a281a9"}
Nov 03 01:11:31.844905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.844528    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm" event={"ID":"c0371001-6166-4147-b499-dfaedc21c26a","Type":"ContainerStarted","Data":"e955c6e081e8e85f6dbfa535b0f4a20337d8152d3ff71b880189367cddba2a90"}
Nov 03 01:11:31.886404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.886344    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm" podStartSLOduration=1.8863312909999999 podStartE2EDuration="1.886331291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:31.883458538 +0000 UTC m=+3974.204995335" watchObservedRunningTime="2026-11-03 01:11:31.886331291 +0000 UTC m=+3974.207868087"
Nov 03 01:11:31.917082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:31.917038    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:32.611404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:32.611364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:32.852371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:32.852282    4550 generic.go:356] "Generic (PLEG): container finished" podID="c0371001-6166-4147-b499-dfaedc21c26a" containerID="ddf55d0803314d0fcaafa46b22f197d653692bc99d140761786b661734a281a9" exitCode=0
Nov 03 01:11:32.852371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:32.852359    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm" event={"ID":"c0371001-6166-4147-b499-dfaedc21c26a","Type":"ContainerDied","Data":"ddf55d0803314d0fcaafa46b22f197d653692bc99d140761786b661734a281a9"}
Nov 03 01:11:33.036730 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:33.036688    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"]
Nov 03 01:11:33.039745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:33.039712    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7a66913-dv5fg"]
Nov 03 01:11:33.054449 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:33.054410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:33.808199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:33.808163    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc62ca5-e313-477d-8496-33a5fed12477" path="/var/lib/kubelet/pods/7fc62ca5-e313-477d-8496-33a5fed12477/volumes"
Nov 03 01:11:34.036657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.036628    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:34.219978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.219940    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k\" (UniqueName: \"kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k\") pod \"c0371001-6166-4147-b499-dfaedc21c26a\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") "
Nov 03 01:11:34.220147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.220010    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca\") pod \"c0371001-6166-4147-b499-dfaedc21c26a\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") "
Nov 03 01:11:34.220147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.220109    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca\") pod \"c0371001-6166-4147-b499-dfaedc21c26a\" (UID: \"c0371001-6166-4147-b499-dfaedc21c26a\") "
Nov 03 01:11:34.220409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.220383    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca" pod "c0371001-6166-4147-b499-dfaedc21c26a" (UID: "c0371001-6166-4147-b499-dfaedc21c26a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:34.220642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.220627    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:34.222085 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.222064    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k" pod "c0371001-6166-4147-b499-dfaedc21c26a" (UID: "c0371001-6166-4147-b499-dfaedc21c26a"). InnerVolumeSpecName "kube-api-access-ttc7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:34.222166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.222104    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca" pod "c0371001-6166-4147-b499-dfaedc21c26a" (UID: "c0371001-6166-4147-b499-dfaedc21c26a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:34.321919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.321876    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttc7k\" (UniqueName: \"kubernetes.io/projected/c0371001-6166-4147-b499-dfaedc21c26a-kube-api-access-ttc7k\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:34.321919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.321910    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0371001-6166-4147-b499-dfaedc21c26a-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:34.829537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.829500    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:34.869742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.869713    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"
Nov 03 01:11:34.869950 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.869740    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm" event={"ID":"c0371001-6166-4147-b499-dfaedc21c26a","Type":"ContainerDied","Data":"e955c6e081e8e85f6dbfa535b0f4a20337d8152d3ff71b880189367cddba2a90"}
Nov 03 01:11:34.869950 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:34.869802    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e955c6e081e8e85f6dbfa535b0f4a20337d8152d3ff71b880189367cddba2a90"
Nov 03 01:11:35.236953 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.236927    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 01:11:35.334167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334128    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334225    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334327    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334414    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334467    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334506    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:35.334694 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334529    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334694 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334593    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334694 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334661    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.334849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334724    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:35.334849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.334734    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:35.335157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.335134    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:35.335238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.335160    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.335238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.335181    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.335238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.335197    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.335238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.335176    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:35.336622 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.336601    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:11:35.336937 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.336917    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:35.336988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.336971    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:35.400716 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.400684    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"]
Nov 03 01:11:35.435785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.435743    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm\") pod \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\" (UID: \"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217\") "
Nov 03 01:11:35.436310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.436290    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.436310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.436312    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.436473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.436325    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.436473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.436336    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.436473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.436344    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.437870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.437852    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm" pod "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" (UID: "36eeeeee-b6cb-4aa4-b3ce-bb59f6588217"). InnerVolumeSpecName "kube-api-access-twptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:35.494497 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.494457    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:35.537604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.537569    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twptm\" (UniqueName: \"kubernetes.io/projected/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217-kube-api-access-twptm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:35.879534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.879500    4550 generic.go:356] "Generic (PLEG): container finished" podID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerID="d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3" exitCode=0
Nov 03 01:11:35.879741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.879570    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"
Nov 03 01:11:35.879741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.879569    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" event={"ID":"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217","Type":"ContainerDied","Data":"d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3"}
Nov 03 01:11:35.879741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.879693    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2" event={"ID":"36eeeeee-b6cb-4aa4-b3ce-bb59f6588217","Type":"ContainerDied","Data":"fa64e6be7d0ae0ad317d252e6a5dba54e24ceb69c87dd9d59956b4cbf61a6d51"}
Nov 03 01:11:35.879741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.879720    4550 scope.go:122] "RemoveContainer" containerID="d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3"
Nov 03 01:11:35.890578 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.890553    4550 scope.go:122] "RemoveContainer" containerID="618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"
Nov 03 01:11:35.899845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.899817    4550 scope.go:122] "RemoveContainer" containerID="d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3"
Nov 03 01:11:35.900125 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:11:35.900106    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3\": container with ID starting with d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3 not found: ID does not exist" containerID="d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3"
Nov 03 01:11:35.900230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.900137    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3"} err="failed to get container status \"d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3\": rpc error: code = NotFound desc = could not find container \"d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3\": container with ID starting with d666a126e5942a20a7dee80e355e09ec62e146f807185afcb890bc6ed665f7a3 not found: ID does not exist"
Nov 03 01:11:35.900230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.900159    4550 scope.go:122] "RemoveContainer" containerID="618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"
Nov 03 01:11:35.900419 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:11:35.900398    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe\": container with ID starting with 618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe not found: ID does not exist" containerID="618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"
Nov 03 01:11:35.900490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.900427    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe"} err="failed to get container status \"618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe\": rpc error: code = NotFound desc = could not find container \"618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe\": container with ID starting with 618c6f6d458c627179f79e51ecab35ffd68695f3a3fe387b3ad484784713c1fe not found: ID does not exist"
Nov 03 01:11:35.937618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.937590    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"]
Nov 03 01:11:35.950167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:35.950141    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2858fb-openshift-default-f6599f46f-nc5q2"]
Nov 03 01:11:36.448624 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:36.448582    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:36.817822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:36.817736    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"]
Nov 03 01:11:36.842785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:36.842747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:37.809620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:37.809583    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" path="/var/lib/kubelet/pods/36eeeeee-b6cb-4aa4-b3ce-bb59f6588217/volumes"
Nov 03 01:11:38.141629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:38.141547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:38.254939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:38.254902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:39.522070 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:39.522033    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:40.221661 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:40.221628    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:40.643962 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:40.643877    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:41.735911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:41.735880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:41.846601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:41.846570    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:42.035268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.035172    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"]
Nov 03 01:11:42.048659 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.048626    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc7cc115-4swsd"]
Nov 03 01:11:42.055911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.055880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:42.559360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.559327    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:42.926332 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.926291    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"]
Nov 03 01:11:42.926891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.926582    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy" containerID="cri-o://6ab12a6e50ce9ad382b7c464b96ce1dc5d47e96092ebdb287457e2fdb5737823" gracePeriod=30
Nov 03 01:11:42.936119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:42.935960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:43.808570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:43.808536    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac581c88-6645-4abd-bd78-efcd30d01d9b" path="/var/lib/kubelet/pods/ac581c88-6645-4abd-bd78-efcd30d01d9b/volumes"
Nov 03 01:11:43.999389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:43.999352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:45.900014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:45.899979    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:46.540922 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.540882    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"]
Nov 03 01:11:46.541453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541425    4550 cpu_manager.go:414] "containerMap: removing container" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy"
Nov 03 01:11:46.541521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541453    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy"
Nov 03 01:11:46.541521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541463    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c0371001-6166-4147-b499-dfaedc21c26a" containerName="revoke-keys"
Nov 03 01:11:46.541521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541468    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0371001-6166-4147-b499-dfaedc21c26a" containerName="revoke-keys"
Nov 03 01:11:46.541521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541479    4550 cpu_manager.go:414] "containerMap: removing container" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy"
Nov 03 01:11:46.541521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541487    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy"
Nov 03 01:11:46.541703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541628    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy"
Nov 03 01:11:46.541703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541642    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0371001-6166-4147-b499-dfaedc21c26a" containerName="revoke-keys"
Nov 03 01:11:46.541703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.541651    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="36eeeeee-b6cb-4aa4-b3ce-bb59f6588217" containerName="istio-proxy"
Nov 03 01:11:46.546530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.546505    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.577028 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.573575    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"]
Nov 03 01:11:46.577028 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.574011    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:11:46.577028 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.574089    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:11:46.620840 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.620799    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkkwm\" (UniqueName: \"kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.621006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.620913    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.621054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.621014    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.721908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.721875    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.722096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.722018    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkkwm\" (UniqueName: \"kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.722148 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.722112    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.722658 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.722627    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.724500 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.724475    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.743040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.743013    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkkwm\" (UniqueName: \"kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm\") pod \"maas-api-revoke-sub-5eb0e557-7d8pv\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:46.747104 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.747069    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:46.858947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.858850    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:46.860665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:46.860644    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:47.214451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:47.211620    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"]
Nov 03 01:11:47.244360 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:11:47.242174    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503bac80_ef28_4562_82eb_f8ceed7ef3d6.slice/crio-215560c7d9e2350a00a01eafa0f00833c391dc136421cdd4bc5ca8b6b230edf3 WatchSource:0}: Error finding container 215560c7d9e2350a00a01eafa0f00833c391dc136421cdd4bc5ca8b6b230edf3: Status 404 returned error can't find the container with id 215560c7d9e2350a00a01eafa0f00833c391dc136421cdd4bc5ca8b6b230edf3
Nov 03 01:11:47.276677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:47.276649    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:47.775785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:47.775699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:48.174389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.173977    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv" event={"ID":"503bac80-ef28-4562-82eb-f8ceed7ef3d6","Type":"ContainerStarted","Data":"f715ee685132970e1d94f4d153d22ecdff7809b161ac6aad8ab2068e0546e12f"}
Nov 03 01:11:48.174389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.174046    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv" event={"ID":"503bac80-ef28-4562-82eb-f8ceed7ef3d6","Type":"ContainerStarted","Data":"215560c7d9e2350a00a01eafa0f00833c391dc136421cdd4bc5ca8b6b230edf3"}
Nov 03 01:11:48.177071 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.177041    4550 generic.go:356] "Generic (PLEG): container finished" podID="a55fb772-8e34-46c9-848d-508b2c44d857" containerID="6ab12a6e50ce9ad382b7c464b96ce1dc5d47e96092ebdb287457e2fdb5737823" exitCode=0
Nov 03 01:11:48.177317 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.177292    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" event={"ID":"a55fb772-8e34-46c9-848d-508b2c44d857","Type":"ContainerDied","Data":"6ab12a6e50ce9ad382b7c464b96ce1dc5d47e96092ebdb287457e2fdb5737823"}
Nov 03 01:11:48.177402 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.177355    4550 scope.go:122] "RemoveContainer" containerID="04090e33e1f66e6cc4c52d5a0fd4f8691cae2e58f62a43cab77fc2e537274fc8"
Nov 03 01:11:48.218187 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.218133    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv" podStartSLOduration=2.218117595 podStartE2EDuration="2.218117595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:48.216701875 +0000 UTC m=+3990.538238676" watchObservedRunningTime="2026-11-03 01:11:48.218117595 +0000 UTC m=+3990.539654411"
Nov 03 01:11:48.271054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.271028    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:11:48.359620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359526    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.359620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359573    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.359864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359639    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.359864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359769    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.359864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359806    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:48.359864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359814    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.360092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359875    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.360092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.359951    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.360092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.360039    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.360092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.360064    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:48.360092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.360069    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token\") pod \"a55fb772-8e34-46c9-848d-508b2c44d857\" (UID: \"a55fb772-8e34-46c9-848d-508b2c44d857\") "
Nov 03 01:11:48.361136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.360726    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.361136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.360752    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.361136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.360724    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:48.361136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.361008    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:48.361136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.361088    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:48.363148 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.362783    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:48.363496 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.363459    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:11:48.363737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.363697    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "kube-api-access-gbxc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:48.364785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.364756    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy" pod "a55fb772-8e34-46c9-848d-508b2c44d857" (UID: "a55fb772-8e34-46c9-848d-508b2c44d857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:48.461759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461717    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.461759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461756    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a55fb772-8e34-46c9-848d-508b2c44d857-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.461979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461772    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxc9\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-kube-api-access-gbxc9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.461979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461787    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a55fb772-8e34-46c9-848d-508b2c44d857-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.461979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461800    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55fb772-8e34-46c9-848d-508b2c44d857-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.461979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461813    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:48.461979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:48.461826    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a55fb772-8e34-46c9-848d-508b2c44d857-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:49.028342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.028246    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"]
Nov 03 01:11:49.040170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.040135    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869af6fd-qtltp"]
Nov 03 01:11:49.186246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.186213    4550 generic.go:356] "Generic (PLEG): container finished" podID="503bac80-ef28-4562-82eb-f8ceed7ef3d6" containerID="f715ee685132970e1d94f4d153d22ecdff7809b161ac6aad8ab2068e0546e12f" exitCode=0
Nov 03 01:11:49.186478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.186323    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv" event={"ID":"503bac80-ef28-4562-82eb-f8ceed7ef3d6","Type":"ContainerDied","Data":"f715ee685132970e1d94f4d153d22ecdff7809b161ac6aad8ab2068e0546e12f"}
Nov 03 01:11:49.187811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.187787    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9" event={"ID":"a55fb772-8e34-46c9-848d-508b2c44d857","Type":"ContainerDied","Data":"49616f0d02712e5c4a93b0b77fa97148ec67f5517fb7762097cd0fb5e23344a6"}
Nov 03 01:11:49.187960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.187832    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"
Nov 03 01:11:49.187960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.187898    4550 scope.go:122] "RemoveContainer" containerID="6ab12a6e50ce9ad382b7c464b96ce1dc5d47e96092ebdb287457e2fdb5737823"
Nov 03 01:11:49.268536 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.268496    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"]
Nov 03 01:11:49.271707 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.271641    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c6e808d7-openshift-default-7969d758df-nrqp9"]
Nov 03 01:11:49.422918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.422878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:49.809439 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.809348    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e08195-7ad1-4b3f-9a8d-d57288aea242" path="/var/lib/kubelet/pods/33e08195-7ad1-4b3f-9a8d-d57288aea242/volumes"
Nov 03 01:11:49.809884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:49.809870    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" path="/var/lib/kubelet/pods/a55fb772-8e34-46c9-848d-508b2c44d857/volumes"
Nov 03 01:11:50.365766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.365734    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:50.422323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.422281    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" containerID="cri-o://b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff" gracePeriod=45
Nov 03 01:11:50.491110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.491037    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm\" (UniqueName: \"kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm\") pod \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") "
Nov 03 01:11:50.491272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.491145    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca\") pod \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") "
Nov 03 01:11:50.491272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.491215    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca\") pod \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\" (UID: \"503bac80-ef28-4562-82eb-f8ceed7ef3d6\") "
Nov 03 01:11:50.491616 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.491582    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca" pod "503bac80-ef28-4562-82eb-f8ceed7ef3d6" (UID: "503bac80-ef28-4562-82eb-f8ceed7ef3d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:50.491778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.491750    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.493293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.493270    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm" pod "503bac80-ef28-4562-82eb-f8ceed7ef3d6" (UID: "503bac80-ef28-4562-82eb-f8ceed7ef3d6"). InnerVolumeSpecName "kube-api-access-fkkwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:50.493758 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.493736    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca" pod "503bac80-ef28-4562-82eb-f8ceed7ef3d6" (UID: "503bac80-ef28-4562-82eb-f8ceed7ef3d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:50.563291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.563242    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": dial tcp 10.133.0.43:8000: connect: connection refused"
Nov 03 01:11:50.592965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.592929    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkkwm\" (UniqueName: \"kubernetes.io/projected/503bac80-ef28-4562-82eb-f8ceed7ef3d6-kube-api-access-fkkwm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.592965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.592962    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/503bac80-ef28-4562-82eb-f8ceed7ef3d6-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.696279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.696244    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 01:11:50.796783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.796693    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs\" (UniqueName: \"kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs\") pod \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") "
Nov 03 01:11:50.796783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.796777    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd\" (UniqueName: \"kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd\") pod \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") "
Nov 03 01:11:50.797013 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.796958    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache\") pod \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") "
Nov 03 01:11:50.797549 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.797522    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache" pod "e697d77c-8549-49d9-b5c1-2f68e6ff53f3" (UID: "e697d77c-8549-49d9-b5c1-2f68e6ff53f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.800848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.800805    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs" pod "e697d77c-8549-49d9-b5c1-2f68e6ff53f3" (UID: "e697d77c-8549-49d9-b5c1-2f68e6ff53f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:50.810822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.810783    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd" pod "e697d77c-8549-49d9-b5c1-2f68e6ff53f3" (UID: "e697d77c-8549-49d9-b5c1-2f68e6ff53f3"). InnerVolumeSpecName "kube-api-access-ftxfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:50.897812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.897776    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm\") pod \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") "
Nov 03 01:11:50.898004 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.897905    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home\") pod \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") "
Nov 03 01:11:50.898120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898106    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir\") pod \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\" (UID: \"e697d77c-8549-49d9-b5c1-2f68e6ff53f3\") "
Nov 03 01:11:50.898223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898195    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home" pod "e697d77c-8549-49d9-b5c1-2f68e6ff53f3" (UID: "e697d77c-8549-49d9-b5c1-2f68e6ff53f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.898413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898387    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir" pod "e697d77c-8549-49d9-b5c1-2f68e6ff53f3" (UID: "e697d77c-8549-49d9-b5c1-2f68e6ff53f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.898891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898873    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.898979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898894    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.898979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898909    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.898979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898924    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftxfd\" (UniqueName: \"kubernetes.io/projected/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-kube-api-access-ftxfd\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.898979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.898936    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.899872 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:50.899849    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm" pod "e697d77c-8549-49d9-b5c1-2f68e6ff53f3" (UID: "e697d77c-8549-49d9-b5c1-2f68e6ff53f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:51.000741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.000701    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e697d77c-8549-49d9-b5c1-2f68e6ff53f3-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:51.067755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.067657    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:51.207631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.207583    4550 generic.go:356] "Generic (PLEG): container finished" podID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerID="b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff" exitCode=0
Nov 03 01:11:51.207883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.207661    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"
Nov 03 01:11:51.207883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.207713    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" event={"ID":"e697d77c-8549-49d9-b5c1-2f68e6ff53f3","Type":"ContainerDied","Data":"b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff"}
Nov 03 01:11:51.207883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.207757    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6" event={"ID":"e697d77c-8549-49d9-b5c1-2f68e6ff53f3","Type":"ContainerDied","Data":"70f62909058d7af4bf0740ad55e9a8130928dbeecc4d599487333bece6c67830"}
Nov 03 01:11:51.207883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.207780    4550 scope.go:122] "RemoveContainer" containerID="b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff"
Nov 03 01:11:51.209525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.209341    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv" event={"ID":"503bac80-ef28-4562-82eb-f8ceed7ef3d6","Type":"ContainerDied","Data":"215560c7d9e2350a00a01eafa0f00833c391dc136421cdd4bc5ca8b6b230edf3"}
Nov 03 01:11:51.209525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.209361    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"
Nov 03 01:11:51.209525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.209416    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="215560c7d9e2350a00a01eafa0f00833c391dc136421cdd4bc5ca8b6b230edf3"
Nov 03 01:11:51.222906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.222887    4550 scope.go:122] "RemoveContainer" containerID="b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff"
Nov 03 01:11:51.223230 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:11:51.223211    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff\": container with ID starting with b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff not found: ID does not exist" containerID="b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff"
Nov 03 01:11:51.223283 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.223237    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff"} err="failed to get container status \"b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff\": rpc error: code = NotFound desc = could not find container \"b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff\": container with ID starting with b1da1b7117a393de9d16e2df97711e0e7edc7c38bfa60df46f45762b0561f3ff not found: ID does not exist"
Nov 03 01:11:51.276932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.276902    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"]
Nov 03 01:11:51.279710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.279682    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/premium-w3-8e90c5-kserve-7576f576df-zwrj6"]
Nov 03 01:11:51.810448 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.810399    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" path="/var/lib/kubelet/pods/e697d77c-8549-49d9-b5c1-2f68e6ff53f3/volumes"
Nov 03 01:11:51.844577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.844535    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" podUID="a6605688-10aa-4a3f-9601-283f0350e94a" containerName="main" containerID="cri-o://01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c" gracePeriod=45
Nov 03 01:11:51.964493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:51.963495    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" podUID="a6605688-10aa-4a3f-9601-283f0350e94a" containerName="main" probeResult="failure" output="Get \"https://10.133.1.27:8000/ready\": dial tcp 10.133.1.27:8000: connect: connection refused"
Nov 03 01:11:52.144396 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.144366    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214571    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7\" (UniqueName: \"kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7\") pod \"a6605688-10aa-4a3f-9601-283f0350e94a\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") "
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214642    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache\") pod \"a6605688-10aa-4a3f-9601-283f0350e94a\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") "
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214717    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs\") pod \"a6605688-10aa-4a3f-9601-283f0350e94a\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") "
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214750    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir\") pod \"a6605688-10aa-4a3f-9601-283f0350e94a\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") "
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214790    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home\") pod \"a6605688-10aa-4a3f-9601-283f0350e94a\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") "
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214829    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm\") pod \"a6605688-10aa-4a3f-9601-283f0350e94a\" (UID: \"a6605688-10aa-4a3f-9601-283f0350e94a\") "
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.214956    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache" pod "a6605688-10aa-4a3f-9601-283f0350e94a" (UID: "a6605688-10aa-4a3f-9601-283f0350e94a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.215188    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir" pod "a6605688-10aa-4a3f-9601-283f0350e94a" (UID: "a6605688-10aa-4a3f-9601-283f0350e94a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.215504    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.215528    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.217471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.215634    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home" pod "a6605688-10aa-4a3f-9601-283f0350e94a" (UID: "a6605688-10aa-4a3f-9601-283f0350e94a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:52.218274 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.217619    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs" pod "a6605688-10aa-4a3f-9601-283f0350e94a" (UID: "a6605688-10aa-4a3f-9601-283f0350e94a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:52.218511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.218482    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7" pod "a6605688-10aa-4a3f-9601-283f0350e94a" (UID: "a6605688-10aa-4a3f-9601-283f0350e94a"). InnerVolumeSpecName "kube-api-access-v4pw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:52.220166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.220145    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm" pod "a6605688-10aa-4a3f-9601-283f0350e94a" (UID: "a6605688-10aa-4a3f-9601-283f0350e94a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:52.222491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.221960    4550 generic.go:356] "Generic (PLEG): container finished" podID="a6605688-10aa-4a3f-9601-283f0350e94a" containerID="01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c" exitCode=0
Nov 03 01:11:52.222491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.222008    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"
Nov 03 01:11:52.222491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.222022    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" event={"ID":"a6605688-10aa-4a3f-9601-283f0350e94a","Type":"ContainerDied","Data":"01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c"}
Nov 03 01:11:52.222491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.222068    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv" event={"ID":"a6605688-10aa-4a3f-9601-283f0350e94a","Type":"ContainerDied","Data":"f6c0edfcbf99ff0fd3f03656b9d2071d06791489b552e033d4aae2a0340fc3d3"}
Nov 03 01:11:52.222491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.222090    4550 scope.go:122] "RemoveContainer" containerID="01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c"
Nov 03 01:11:52.241648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.241626    4550 scope.go:122] "RemoveContainer" containerID="01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c"
Nov 03 01:11:52.241966 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:11:52.241942    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c\": container with ID starting with 01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c not found: ID does not exist" containerID="01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c"
Nov 03 01:11:52.242012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.241977    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c"} err="failed to get container status \"01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c\": rpc error: code = NotFound desc = could not find container \"01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c\": container with ID starting with 01234c4b76db0c364d54120105bfe67e3ec691d938d4e9fb3c367dbf5ed8d93c not found: ID does not exist"
Nov 03 01:11:52.277806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.277766    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"]
Nov 03 01:11:52.281045 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.281016    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/rate-test-model-20109d-kserve-6cf886b8df-pkwkv"]
Nov 03 01:11:52.317031 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.316998    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4pw7\" (UniqueName: \"kubernetes.io/projected/a6605688-10aa-4a3f-9601-283f0350e94a-kube-api-access-v4pw7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.317031 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.317027    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6605688-10aa-4a3f-9601-283f0350e94a-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.317031 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.317038    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.317252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.317045    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6605688-10aa-4a3f-9601-283f0350e94a-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:11:52.839488 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:52.839453    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:53.809443 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:53.809387    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6605688-10aa-4a3f-9601-283f0350e94a" path="/var/lib/kubelet/pods/a6605688-10aa-4a3f-9601-283f0350e94a/volumes"
Nov 03 01:11:53.978279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:53.978242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:54.612482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:54.612275    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:54.726899 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:11:54.726862    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:11:55.895180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:55.895139    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:56.059033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:56.058997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:56.822140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:56.822109    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:58.533015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:58.532979    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:11:58.844088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:11:58.843986    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:00.130247 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:00.130154    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:00.877052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:00.877020    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:01.174936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:01.174887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:02.628348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:02.628310    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:03.395934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:03.395901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:03.512151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:03.512101    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:04.190553 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:04.190503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:04.442772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:04.442695    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:04.620996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:04.620962    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:05.486678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.486640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:05.947537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.947501    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"]
Nov 03 01:12:05.948003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.947990    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a6605688-10aa-4a3f-9601-283f0350e94a" containerName="main"
Nov 03 01:12:05.948047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948004    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6605688-10aa-4a3f-9601-283f0350e94a" containerName="main"
Nov 03 01:12:05.948047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948015    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy"
Nov 03 01:12:05.948047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948021    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy"
Nov 03 01:12:05.948047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948031    4550 cpu_manager.go:414] "containerMap: removing container" podUID="503bac80-ef28-4562-82eb-f8ceed7ef3d6" containerName="revoke-keys"
Nov 03 01:12:05.948047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948036    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="503bac80-ef28-4562-82eb-f8ceed7ef3d6" containerName="revoke-keys"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948067    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948072    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948144    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948155    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6605688-10aa-4a3f-9601-283f0350e94a" containerName="main"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948160    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948165    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="503bac80-ef28-4562-82eb-f8ceed7ef3d6" containerName="revoke-keys"
Nov 03 01:12:05.948210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.948176    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e697d77c-8549-49d9-b5c1-2f68e6ff53f3" containerName="main"
Nov 03 01:12:05.951550 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.951533    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:05.971088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.971059    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"]
Nov 03 01:12:05.979846 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.979823    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:12:05.979994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:05.979900    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:12:06.131163 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.131124    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88qp\" (UniqueName: \"kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.131346 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.131175    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.131346 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.131276    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.232731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.232652    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88qp\" (UniqueName: \"kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.232731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.232686    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.232731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.232716    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.233425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.233399    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.234446 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.234413    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:06.235237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.235216    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.252350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.252327    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88qp\" (UniqueName: \"kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp\") pod \"maas-api-revoke-sub-9c4a9749-bxcck\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.268989 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.268970    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:06.598462 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:12:06.594744    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44f6a52f_4ab3_4b77_bfe2_dfbe45903309.slice/crio-2f65d10ee56ae2cb312b83daaa526ad274ecb74affdd1b258079abd0c7d694f2 WatchSource:0}: Error finding container 2f65d10ee56ae2cb312b83daaa526ad274ecb74affdd1b258079abd0c7d694f2: Status 404 returned error can't find the container with id 2f65d10ee56ae2cb312b83daaa526ad274ecb74affdd1b258079abd0c7d694f2
Nov 03 01:12:06.598462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:06.596414    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"]
Nov 03 01:12:07.225591 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:07.225557    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:07.334781 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:07.334751    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:07.348261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:07.348229    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck" event={"ID":"44f6a52f-4ab3-4b77-bfe2-dfbe45903309","Type":"ContainerStarted","Data":"1827033eceaec0033be3cf437c968ad52d2158da98a7c9cc7b5a8baa004ee86b"}
Nov 03 01:12:07.348403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:07.348306    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck" event={"ID":"44f6a52f-4ab3-4b77-bfe2-dfbe45903309","Type":"ContainerStarted","Data":"2f65d10ee56ae2cb312b83daaa526ad274ecb74affdd1b258079abd0c7d694f2"}
Nov 03 01:12:07.390284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:07.390231    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck" podStartSLOduration=2.390214941 podStartE2EDuration="2.390214941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:07.388586034 +0000 UTC m=+4009.710122828" watchObservedRunningTime="2026-11-03 01:12:07.390214941 +0000 UTC m=+4009.711751741"
Nov 03 01:12:07.939657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:07.939621    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:08.356038 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:08.355944    4550 generic.go:356] "Generic (PLEG): container finished" podID="44f6a52f-4ab3-4b77-bfe2-dfbe45903309" containerID="1827033eceaec0033be3cf437c968ad52d2158da98a7c9cc7b5a8baa004ee86b" exitCode=0
Nov 03 01:12:08.356200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:08.356034    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck" event={"ID":"44f6a52f-4ab3-4b77-bfe2-dfbe45903309","Type":"ContainerDied","Data":"1827033eceaec0033be3cf437c968ad52d2158da98a7c9cc7b5a8baa004ee86b"}
Nov 03 01:12:09.528327 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.528299    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:09.578372 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.578326    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp\" (UniqueName: \"kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp\") pod \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") "
Nov 03 01:12:09.578702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.578676    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca\") pod \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") "
Nov 03 01:12:09.578841 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.578805    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca\") pod \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\" (UID: \"44f6a52f-4ab3-4b77-bfe2-dfbe45903309\") "
Nov 03 01:12:09.579405 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.579305    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca" pod "44f6a52f-4ab3-4b77-bfe2-dfbe45903309" (UID: "44f6a52f-4ab3-4b77-bfe2-dfbe45903309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:09.582316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.582268    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca" pod "44f6a52f-4ab3-4b77-bfe2-dfbe45903309" (UID: "44f6a52f-4ab3-4b77-bfe2-dfbe45903309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:09.585374 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.585351    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp" pod "44f6a52f-4ab3-4b77-bfe2-dfbe45903309" (UID: "44f6a52f-4ab3-4b77-bfe2-dfbe45903309"). InnerVolumeSpecName "kube-api-access-s88qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:09.680715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.680679    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88qp\" (UniqueName: \"kubernetes.io/projected/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-kube-api-access-s88qp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:09.680715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.680715    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:09.680715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.680730    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44f6a52f-4ab3-4b77-bfe2-dfbe45903309-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:09.713242 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:09.713213    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:10.371630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:10.371534    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck" event={"ID":"44f6a52f-4ab3-4b77-bfe2-dfbe45903309","Type":"ContainerDied","Data":"2f65d10ee56ae2cb312b83daaa526ad274ecb74affdd1b258079abd0c7d694f2"}
Nov 03 01:12:10.371630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:10.371592    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"
Nov 03 01:12:10.371630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:10.371608    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f65d10ee56ae2cb312b83daaa526ad274ecb74affdd1b258079abd0c7d694f2"
Nov 03 01:12:11.332525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:11.332491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:11.451137 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:11.451104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:11.556065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:11.556027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:12.916825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:12.916790    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:13.213769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:13.213664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:13.311093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:13.311060    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:13.426788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:13.426749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:13.915717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:13.915675    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:14.397715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:14.397643    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:14.624943 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:14.624909    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:14.949976 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:14.949938    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:15.049236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:15.049198    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:15.442963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:15.442929    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:15.662821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:15.662788    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:16.286233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.286195    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"]
Nov 03 01:12:16.287061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.287041    4550 cpu_manager.go:414] "containerMap: removing container" podUID="44f6a52f-4ab3-4b77-bfe2-dfbe45903309" containerName="revoke-keys"
Nov 03 01:12:16.287120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.287065    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f6a52f-4ab3-4b77-bfe2-dfbe45903309" containerName="revoke-keys"
Nov 03 01:12:16.287120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.287107    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy"
Nov 03 01:12:16.287120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.287114    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55fb772-8e34-46c9-848d-508b2c44d857" containerName="istio-proxy"
Nov 03 01:12:16.287240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.287228    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f6a52f-4ab3-4b77-bfe2-dfbe45903309" containerName="revoke-keys"
Nov 03 01:12:16.291336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.291318    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.307295 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.307266    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"]
Nov 03 01:12:16.315762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.315735    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:12:16.315762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.315745    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:12:16.381882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.381848    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28ph\" (UniqueName: \"kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.382113 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.382054    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.382113 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.382090    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.483680 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.483633    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28ph\" (UniqueName: \"kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.484196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.483760    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.484196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.483785    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.484519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.484491    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.486280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.486255    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.504290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.504259    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28ph\" (UniqueName: \"kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph\") pod \"maas-api-revoke-sub-cedfe99f-fv4qc\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.610292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.610212    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:16.909639 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:12:16.909607    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdb7f84_b258_4727_bdac_b4c6a36a162f.slice/crio-059639a39d98e9502b12ab2cd05032ac7610e82d9440d365d542f93846ccba48 WatchSource:0}: Error finding container 059639a39d98e9502b12ab2cd05032ac7610e82d9440d365d542f93846ccba48: Status 404 returned error can't find the container with id 059639a39d98e9502b12ab2cd05032ac7610e82d9440d365d542f93846ccba48
Nov 03 01:12:16.910093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:16.910067    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"]
Nov 03 01:12:17.107615 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:17.107573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:17.440630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:17.440539    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc" event={"ID":"5cdb7f84-b258-4727-bdac-b4c6a36a162f","Type":"ContainerStarted","Data":"0d513897b2de99e3dbe5c40b2e278ace49b366361b1913048aa63fe05332e994"}
Nov 03 01:12:17.440630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:17.440596    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc" event={"ID":"5cdb7f84-b258-4727-bdac-b4c6a36a162f","Type":"ContainerStarted","Data":"059639a39d98e9502b12ab2cd05032ac7610e82d9440d365d542f93846ccba48"}
Nov 03 01:12:17.483646 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:17.483569    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc" podStartSLOduration=1.483548048 podStartE2EDuration="1.483548048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:17.48141882 +0000 UTC m=+4019.802955641" watchObservedRunningTime="2026-11-03 01:12:17.483548048 +0000 UTC m=+4019.805084849"
Nov 03 01:12:18.448262 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:18.448220    4550 generic.go:356] "Generic (PLEG): container finished" podID="5cdb7f84-b258-4727-bdac-b4c6a36a162f" containerID="0d513897b2de99e3dbe5c40b2e278ace49b366361b1913048aa63fe05332e994" exitCode=0
Nov 03 01:12:18.448710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:18.448266    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc" event={"ID":"5cdb7f84-b258-4727-bdac-b4c6a36a162f","Type":"ContainerDied","Data":"0d513897b2de99e3dbe5c40b2e278ace49b366361b1913048aa63fe05332e994"}
Nov 03 01:12:18.469084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:18.469036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:18.718168 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:18.718068    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:19.631068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.631041    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:19.739678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.739600    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph\" (UniqueName: \"kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph\") pod \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") "
Nov 03 01:12:19.739678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.739672    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca\") pod \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") "
Nov 03 01:12:19.739947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.739720    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca\") pod \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\" (UID: \"5cdb7f84-b258-4727-bdac-b4c6a36a162f\") "
Nov 03 01:12:19.740118 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.740093    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca" pod "5cdb7f84-b258-4727-bdac-b4c6a36a162f" (UID: "5cdb7f84-b258-4727-bdac-b4c6a36a162f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:19.740238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.740218    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:19.741704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.741677    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca" pod "5cdb7f84-b258-4727-bdac-b4c6a36a162f" (UID: "5cdb7f84-b258-4727-bdac-b4c6a36a162f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:19.741862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.741843    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph" pod "5cdb7f84-b258-4727-bdac-b4c6a36a162f" (UID: "5cdb7f84-b258-4727-bdac-b4c6a36a162f"). InnerVolumeSpecName "kube-api-access-f28ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:19.841330 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.841293    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28ph\" (UniqueName: \"kubernetes.io/projected/5cdb7f84-b258-4727-bdac-b4c6a36a162f-kube-api-access-f28ph\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:19.841330 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.841325    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cdb7f84-b258-4727-bdac-b4c6a36a162f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:19.919013 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:19.918975    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:20.424919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:20.424881    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:20.463074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:20.463044    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"
Nov 03 01:12:20.463074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:20.463056    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc" event={"ID":"5cdb7f84-b258-4727-bdac-b4c6a36a162f","Type":"ContainerDied","Data":"059639a39d98e9502b12ab2cd05032ac7610e82d9440d365d542f93846ccba48"}
Nov 03 01:12:20.463296 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:20.463106    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="059639a39d98e9502b12ab2cd05032ac7610e82d9440d365d542f93846ccba48"
Nov 03 01:12:22.340864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:22.340829    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:24.119673 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:24.119632    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:24.725920 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:12:24.725879    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:12:24.871737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:24.871697    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:24.972961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:24.972927    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:25.963590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:25.963554    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:27.651152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:27.651119    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:29.363620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:29.363573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:29.916338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:29.916302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:30.023947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:30.023906    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:31.054284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:31.054248    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:32.705648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:32.705615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:33.697873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:33.697770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:34.438853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:34.438813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:35.202350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:35.202296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:36.004235 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:36.004194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:36.597676 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:36.597639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:36.737125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:36.737086    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:36.872838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:36.872673    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:38.622912 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:38.622874    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:40.390196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:40.390153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:42.079679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:42.079644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:43.717248 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:43.717216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:44.848400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:44.848365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:44.959462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:44.959409    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:45.155872 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:45.155832    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"]
Nov 03 01:12:45.206678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:45.206638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:45.357378 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:45.357340    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:45.764339 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:45.764306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:45.917345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:45.917306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:46.007588 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.007548    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"]
Nov 03 01:12:46.007893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.007826    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy" containerID="cri-o://8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505" gracePeriod=30
Nov 03 01:12:46.035476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.035347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:46.515506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.515472    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:46.969033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.968996    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"]
Nov 03 01:12:46.969837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.969816    4550 cpu_manager.go:414] "containerMap: removing container" podUID="5cdb7f84-b258-4727-bdac-b4c6a36a162f" containerName="revoke-keys"
Nov 03 01:12:46.969837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.969839    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdb7f84-b258-4727-bdac-b4c6a36a162f" containerName="revoke-keys"
Nov 03 01:12:46.970018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.970006    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cdb7f84-b258-4727-bdac-b4c6a36a162f" containerName="revoke-keys"
Nov 03 01:12:46.973902 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.973877    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:46.990863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:46.990820    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"]
Nov 03 01:12:47.031121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.031078    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:12:47.031319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.031276    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:12:47.078366 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.078317    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjrdb\" (UniqueName: \"kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.078627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.078380    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.078627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.078529    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.180070 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.180029    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjrdb\" (UniqueName: \"kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.180070 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.180074    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.180359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.180111    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.180972 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.180945    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.182652 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.182622    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.199967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.199927    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjrdb\" (UniqueName: \"kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb\") pod \"maas-api-revoke-sub-38ef8ba0-l4kv7\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.287233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.287140    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:47.338790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.338754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:47.591656 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.591618    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"]
Nov 03 01:12:47.591939 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:12:47.591914    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f653e3_9e68_428e_b7dc_2b74151e6b75.slice/crio-cd50888208e9ba2a8c4d1eaa1eae59dc6b2a36aeb098bafff5018f49f6b6c75b WatchSource:0}: Error finding container cd50888208e9ba2a8c4d1eaa1eae59dc6b2a36aeb098bafff5018f49f6b6c75b: Status 404 returned error can't find the container with id cd50888208e9ba2a8c4d1eaa1eae59dc6b2a36aeb098bafff5018f49f6b6c75b
Nov 03 01:12:47.652486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:47.652454    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7" event={"ID":"f6f653e3-9e68-428e-b7dc-2b74151e6b75","Type":"ContainerStarted","Data":"cd50888208e9ba2a8c4d1eaa1eae59dc6b2a36aeb098bafff5018f49f6b6c75b"}
Nov 03 01:12:48.662551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:48.662503    4550 generic.go:356] "Generic (PLEG): container finished" podID="f6f653e3-9e68-428e-b7dc-2b74151e6b75" containerID="ad4b65603776a8cef6ffa9fd7a9ecc40546321fd09f612c474c0c2ae38ffb832" exitCode=0
Nov 03 01:12:48.662965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:48.662689    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7" event={"ID":"f6f653e3-9e68-428e-b7dc-2b74151e6b75","Type":"ContainerDied","Data":"ad4b65603776a8cef6ffa9fd7a9ecc40546321fd09f612c474c0c2ae38ffb832"}
Nov 03 01:12:48.988280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:48.988236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:49.842936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.842907    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:49.924109 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.924002    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca\") pod \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") "
Nov 03 01:12:49.924306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.924203    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb\" (UniqueName: \"kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb\") pod \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") "
Nov 03 01:12:49.924306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.924256    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca\") pod \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\" (UID: \"f6f653e3-9e68-428e-b7dc-2b74151e6b75\") "
Nov 03 01:12:49.925543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.925512    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca" pod "f6f653e3-9e68-428e-b7dc-2b74151e6b75" (UID: "f6f653e3-9e68-428e-b7dc-2b74151e6b75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:49.926865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.926835    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca" pod "f6f653e3-9e68-428e-b7dc-2b74151e6b75" (UID: "f6f653e3-9e68-428e-b7dc-2b74151e6b75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:49.927507 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:49.927467    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb" pod "f6f653e3-9e68-428e-b7dc-2b74151e6b75" (UID: "f6f653e3-9e68-428e-b7dc-2b74151e6b75"). InnerVolumeSpecName "kube-api-access-zjrdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:50.026745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.026705    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.026745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.026739    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjrdb\" (UniqueName: \"kubernetes.io/projected/f6f653e3-9e68-428e-b7dc-2b74151e6b75-kube-api-access-zjrdb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.026745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.026749    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f653e3-9e68-428e-b7dc-2b74151e6b75-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.671030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.670991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:50.678898 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.678868    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"
Nov 03 01:12:50.679100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.679072    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7" event={"ID":"f6f653e3-9e68-428e-b7dc-2b74151e6b75","Type":"ContainerDied","Data":"cd50888208e9ba2a8c4d1eaa1eae59dc6b2a36aeb098bafff5018f49f6b6c75b"}
Nov 03 01:12:50.679186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:50.679136    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd50888208e9ba2a8c4d1eaa1eae59dc6b2a36aeb098bafff5018f49f6b6c75b"
Nov 03 01:12:51.306103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.306065    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 01:12:51.344687 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344655    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.344869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344697    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.344869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344730    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.344869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344761    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.344869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344839    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.345063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344908    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.345063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344958    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.345063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.344978    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:51.345063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345031    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.345302 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345090    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:51.345302 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345165    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:51.345302 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345173    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo\") pod \"a825147e-2d97-4a2c-b305-24b2b19c6487\" (UID: \"a825147e-2d97-4a2c-b305-24b2b19c6487\") "
Nov 03 01:12:51.345518 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345335    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:51.345518 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345412    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:51.345886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345857    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.346012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345890    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.346012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345906    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.346012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345919    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a825147e-2d97-4a2c-b305-24b2b19c6487-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.346012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.345932    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.347202 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.347177    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "kube-api-access-7cbp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:51.347324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.347217    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:51.347324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.347223    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:51.347483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.347462    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo" pod "a825147e-2d97-4a2c-b305-24b2b19c6487" (UID: "a825147e-2d97-4a2c-b305-24b2b19c6487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:12:51.447041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.447005    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.447041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.447035    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.447041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.447045    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.447300 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.447053    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbp4\" (UniqueName: \"kubernetes.io/projected/a825147e-2d97-4a2c-b305-24b2b19c6487-kube-api-access-7cbp4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:12:51.480456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.480395    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"]
Nov 03 01:12:51.501917 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.501887    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"]
Nov 03 01:12:51.598876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.598774    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:51.686635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.686601    4550 generic.go:356] "Generic (PLEG): container finished" podID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerID="8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505" exitCode=0
Nov 03 01:12:51.686865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.686671    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" event={"ID":"a825147e-2d97-4a2c-b305-24b2b19c6487","Type":"ContainerDied","Data":"8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505"}
Nov 03 01:12:51.686865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.686700    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"
Nov 03 01:12:51.686865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.686769    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq" event={"ID":"a825147e-2d97-4a2c-b305-24b2b19c6487","Type":"ContainerDied","Data":"c9cdc1df5e32493de8fa3affdda5137255ea0b66d56bc6da71bfaadea5669cc4"}
Nov 03 01:12:51.686865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.686801    4550 scope.go:122] "RemoveContainer" containerID="8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505"
Nov 03 01:12:51.698278 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.698195    4550 scope.go:122] "RemoveContainer" containerID="8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505"
Nov 03 01:12:51.698720 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:12:51.698692    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505\": container with ID starting with 8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505 not found: ID does not exist" containerID="8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505"
Nov 03 01:12:51.698845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.698730    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505"} err="failed to get container status \"8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505\": rpc error: code = NotFound desc = could not find container \"8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505\": container with ID starting with 8976794b009586e57cbb7aec49ed00e42d7cdf41e2f4a06f7a996704723fd505 not found: ID does not exist"
Nov 03 01:12:51.742787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.742755    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"]
Nov 03 01:12:51.751167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.751134    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a07a90-openshift-default-7bfb466c97-b6rwq"]
Nov 03 01:12:51.808468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:51.808421    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" path="/var/lib/kubelet/pods/a825147e-2d97-4a2c-b305-24b2b19c6487/volumes"
Nov 03 01:12:52.547071 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:52.547000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:52.862607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:52.862410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:52.962025 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:52.961985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:53.814501 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:53.814463    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:54.328220 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:54.328176    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:54.724131 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:12:54.724090    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:12:55.921308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:55.921270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:57.291537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:57.291495    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:57.696152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:57.696111    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:59.246062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:59.246023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:12:59.371597 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:12:59.371562    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:00.178337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.178282    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" podUID="6d4e9365-33fb-4342-8e09-1923fde1c424" containerName="main" containerID="cri-o://79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4" gracePeriod=45
Nov 03 01:13:00.469049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.469025    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:13:00.580628 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580593    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir\") pod \"6d4e9365-33fb-4342-8e09-1923fde1c424\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") "
Nov 03 01:13:00.580815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580657    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home\") pod \"6d4e9365-33fb-4342-8e09-1923fde1c424\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") "
Nov 03 01:13:00.580815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580730    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v\" (UniqueName: \"kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v\") pod \"6d4e9365-33fb-4342-8e09-1923fde1c424\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") "
Nov 03 01:13:00.581008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580830    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache\") pod \"6d4e9365-33fb-4342-8e09-1923fde1c424\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") "
Nov 03 01:13:00.581008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580856    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm\") pod \"6d4e9365-33fb-4342-8e09-1923fde1c424\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") "
Nov 03 01:13:00.581008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580859    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir" pod "6d4e9365-33fb-4342-8e09-1923fde1c424" (UID: "6d4e9365-33fb-4342-8e09-1923fde1c424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:00.581008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580932    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home" pod "6d4e9365-33fb-4342-8e09-1923fde1c424" (UID: "6d4e9365-33fb-4342-8e09-1923fde1c424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:00.581008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.580965    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs\") pod \"6d4e9365-33fb-4342-8e09-1923fde1c424\" (UID: \"6d4e9365-33fb-4342-8e09-1923fde1c424\") "
Nov 03 01:13:00.581249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.581104    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache" pod "6d4e9365-33fb-4342-8e09-1923fde1c424" (UID: "6d4e9365-33fb-4342-8e09-1923fde1c424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:00.581509 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.581485    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:00.581707 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.581686    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:00.581786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.581712    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:00.583129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.583110    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm" pod "6d4e9365-33fb-4342-8e09-1923fde1c424" (UID: "6d4e9365-33fb-4342-8e09-1923fde1c424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:00.583211 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.583155    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs" pod "6d4e9365-33fb-4342-8e09-1923fde1c424" (UID: "6d4e9365-33fb-4342-8e09-1923fde1c424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:00.583292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.583274    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v" pod "6d4e9365-33fb-4342-8e09-1923fde1c424" (UID: "6d4e9365-33fb-4342-8e09-1923fde1c424"). InnerVolumeSpecName "kube-api-access-6cq2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:00.682744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.682706    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cq2v\" (UniqueName: \"kubernetes.io/projected/6d4e9365-33fb-4342-8e09-1923fde1c424-kube-api-access-6cq2v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:00.682744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.682735    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d4e9365-33fb-4342-8e09-1923fde1c424-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:00.682744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.682746    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d4e9365-33fb-4342-8e09-1923fde1c424-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:00.753024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.752935    4550 generic.go:356] "Generic (PLEG): container finished" podID="6d4e9365-33fb-4342-8e09-1923fde1c424" containerID="79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4" exitCode=0
Nov 03 01:13:00.753024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.752995    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"
Nov 03 01:13:00.753230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.753022    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" event={"ID":"6d4e9365-33fb-4342-8e09-1923fde1c424","Type":"ContainerDied","Data":"79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4"}
Nov 03 01:13:00.753230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.753076    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" event={"ID":"6d4e9365-33fb-4342-8e09-1923fde1c424","Type":"ContainerDied","Data":"d83f6c13504b4dd0dd90a507a7f1a06c767620ed6fa7d4b87f64a33441ca5156"}
Nov 03 01:13:00.753230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.753099    4550 scope.go:122] "RemoveContainer" containerID="79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4"
Nov 03 01:13:00.763665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.763645    4550 scope.go:122] "RemoveContainer" containerID="79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4"
Nov 03 01:13:00.763965 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:00.763938    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4\": container with ID starting with 79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4 not found: ID does not exist" containerID="79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4"
Nov 03 01:13:00.764064 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.763970    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4"} err="failed to get container status \"79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4\": rpc error: code = NotFound desc = could not find container \"79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4\": container with ID starting with 79db94b4f210502c3d3c9a376f731945ba0ae790bb1aaf90cd5521e8278e32a4 not found: ID does not exist"
Nov 03 01:13:00.828412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.828374    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"]
Nov 03 01:13:00.841812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.841772    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw"]
Nov 03 01:13:00.913268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:00.913233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:01.418771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:01.418724    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/rate-test-model-7ef1b6-kserve-69b64885cc-48jxw" podUID="6d4e9365-33fb-4342-8e09-1923fde1c424" containerName="main" probeResult="failure" output="Get \"https://10.133.1.29:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:13:01.809049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:01.808962    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d4e9365-33fb-4342-8e09-1923fde1c424" path="/var/lib/kubelet/pods/6d4e9365-33fb-4342-8e09-1923fde1c424/volumes"
Nov 03 01:13:02.367513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.367479    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"]
Nov 03 01:13:02.368063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368043    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f6f653e3-9e68-428e-b7dc-2b74151e6b75" containerName="revoke-keys"
Nov 03 01:13:02.368169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368066    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f653e3-9e68-428e-b7dc-2b74151e6b75" containerName="revoke-keys"
Nov 03 01:13:02.368169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368098    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy"
Nov 03 01:13:02.368169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368107    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy"
Nov 03 01:13:02.368169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368141    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6d4e9365-33fb-4342-8e09-1923fde1c424" containerName="main"
Nov 03 01:13:02.368169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368148    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d4e9365-33fb-4342-8e09-1923fde1c424" containerName="main"
Nov 03 01:13:02.368454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368292    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d4e9365-33fb-4342-8e09-1923fde1c424" containerName="main"
Nov 03 01:13:02.368454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368312    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f653e3-9e68-428e-b7dc-2b74151e6b75" containerName="revoke-keys"
Nov 03 01:13:02.368454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.368323    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a825147e-2d97-4a2c-b305-24b2b19c6487" containerName="istio-proxy"
Nov 03 01:13:02.372987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.372965    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.388978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.388950    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"]
Nov 03 01:13:02.398460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.397949    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:13:02.398460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.397990    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:13:02.506351 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.506305    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkh8\" (UniqueName: \"kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.506582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.506449    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.506582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.506533    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.548307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.548276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:02.608618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.608573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.608865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.608774    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkh8\" (UniqueName: \"kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.608961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.608869    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.609769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.609740    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.612585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.612559    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.631032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.630941    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkh8\" (UniqueName: \"kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8\") pod \"maas-api-revoke-sub-42dab15d-dd69l\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.687642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.687606    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:02.992135 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:13:02.992102    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee38c88b_5480_49c0_a796_d4c37df8e133.slice/crio-94543263459decb4dbfae1b46db2293c3c1b1a0f8f1b13771ee25c7cb1d295dd WatchSource:0}: Error finding container 94543263459decb4dbfae1b46db2293c3c1b1a0f8f1b13771ee25c7cb1d295dd: Status 404 returned error can't find the container with id 94543263459decb4dbfae1b46db2293c3c1b1a0f8f1b13771ee25c7cb1d295dd
Nov 03 01:13:02.993360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:02.993329    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"]
Nov 03 01:13:03.781735 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:03.781696    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l" event={"ID":"ee38c88b-5480-49c0-a796-d4c37df8e133","Type":"ContainerStarted","Data":"dc6f576fc2204fb5041206db5e04da6ab76f7169c494d2760f2567aa21054d21"}
Nov 03 01:13:03.781873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:03.781761    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l" event={"ID":"ee38c88b-5480-49c0-a796-d4c37df8e133","Type":"ContainerStarted","Data":"94543263459decb4dbfae1b46db2293c3c1b1a0f8f1b13771ee25c7cb1d295dd"}
Nov 03 01:13:03.828388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:03.828335    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l" podStartSLOduration=1.828319344 podStartE2EDuration="1.828319344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:03.824849275 +0000 UTC m=+4066.146386074" watchObservedRunningTime="2026-11-03 01:13:03.828319344 +0000 UTC m=+4066.149856146"
Nov 03 01:13:04.334475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:04.334443    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:04.434052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:04.434013    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:04.698953 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:04.698919    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:04.797411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:04.797236    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:04.810792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:04.810750    4550 generic.go:356] "Generic (PLEG): container finished" podID="ee38c88b-5480-49c0-a796-d4c37df8e133" containerID="dc6f576fc2204fb5041206db5e04da6ab76f7169c494d2760f2567aa21054d21" exitCode=0
Nov 03 01:13:04.810995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:04.810837    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l" event={"ID":"ee38c88b-5480-49c0-a796-d4c37df8e133","Type":"ContainerDied","Data":"dc6f576fc2204fb5041206db5e04da6ab76f7169c494d2760f2567aa21054d21"}
Nov 03 01:13:05.988417 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:05.988387    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:06.058621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.058582    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8\" (UniqueName: \"kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8\") pod \"ee38c88b-5480-49c0-a796-d4c37df8e133\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") "
Nov 03 01:13:06.058859 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.058642    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca\") pod \"ee38c88b-5480-49c0-a796-d4c37df8e133\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") "
Nov 03 01:13:06.058859 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.058768    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca\") pod \"ee38c88b-5480-49c0-a796-d4c37df8e133\" (UID: \"ee38c88b-5480-49c0-a796-d4c37df8e133\") "
Nov 03 01:13:06.059474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.059419    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca" pod "ee38c88b-5480-49c0-a796-d4c37df8e133" (UID: "ee38c88b-5480-49c0-a796-d4c37df8e133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:06.060897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.060872    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca" pod "ee38c88b-5480-49c0-a796-d4c37df8e133" (UID: "ee38c88b-5480-49c0-a796-d4c37df8e133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:06.061339 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.061321    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8" pod "ee38c88b-5480-49c0-a796-d4c37df8e133" (UID: "ee38c88b-5480-49c0-a796-d4c37df8e133"). InnerVolumeSpecName "kube-api-access-dlkh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:06.137218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.137170    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:06.160648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.160612    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkh8\" (UniqueName: \"kubernetes.io/projected/ee38c88b-5480-49c0-a796-d4c37df8e133-kube-api-access-dlkh8\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.160648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.160649    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.160863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.160661    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee38c88b-5480-49c0-a796-d4c37df8e133-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.508766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.508729    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" containerID="cri-o://7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6" gracePeriod=45
Nov 03 01:13:06.522115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.522078    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" containerID="cri-o://5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83" gracePeriod=45
Nov 03 01:13:06.567947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.567909    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused"
Nov 03 01:13:06.680349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.680192    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused"
Nov 03 01:13:06.786174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.786133    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 01:13:06.825959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.825926    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 01:13:06.826795 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.826764    4550 generic.go:356] "Generic (PLEG): container finished" podID="776a620a-09d1-44e1-8953-d31795378672" containerID="7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6" exitCode=0
Nov 03 01:13:06.827007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.826820    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"
Nov 03 01:13:06.827007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.826850    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" event={"ID":"776a620a-09d1-44e1-8953-d31795378672","Type":"ContainerDied","Data":"7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6"}
Nov 03 01:13:06.827007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.826924    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj" event={"ID":"776a620a-09d1-44e1-8953-d31795378672","Type":"ContainerDied","Data":"faa60909ef339023d6b8db276a58240bb981444914e62cb1d2f1274ca8dd86dd"}
Nov 03 01:13:06.827007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.826946    4550 scope.go:122] "RemoveContainer" containerID="7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6"
Nov 03 01:13:06.830354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.829555    4550 generic.go:356] "Generic (PLEG): container finished" podID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerID="5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83" exitCode=0
Nov 03 01:13:06.830354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.829622    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"
Nov 03 01:13:06.830354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.829721    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" event={"ID":"a20455af-f5bf-41ca-a942-8c2b6b35d962","Type":"ContainerDied","Data":"5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83"}
Nov 03 01:13:06.830354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.829755    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb" event={"ID":"a20455af-f5bf-41ca-a942-8c2b6b35d962","Type":"ContainerDied","Data":"71fe5109fd31a1fadd88c440689a5d6779cbfb138d803cdff27dd3435a90df09"}
Nov 03 01:13:06.836696 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.836651    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l" event={"ID":"ee38c88b-5480-49c0-a796-d4c37df8e133","Type":"ContainerDied","Data":"94543263459decb4dbfae1b46db2293c3c1b1a0f8f1b13771ee25c7cb1d295dd"}
Nov 03 01:13:06.836817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.836709    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94543263459decb4dbfae1b46db2293c3c1b1a0f8f1b13771ee25c7cb1d295dd"
Nov 03 01:13:06.836817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.836775    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"
Nov 03 01:13:06.841603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.841582    4550 scope.go:122] "RemoveContainer" containerID="7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6"
Nov 03 01:13:06.846114 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:06.844773    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6\": container with ID starting with 7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6 not found: ID does not exist" containerID="7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6"
Nov 03 01:13:06.846114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.844811    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6"} err="failed to get container status \"7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6\": rpc error: code = NotFound desc = could not find container \"7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6\": container with ID starting with 7390ad1f2aa21b6dabc6d6adcad1edc35b6c2ea21e602d1822e71cffeeb3b5b6 not found: ID does not exist"
Nov 03 01:13:06.846114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.844837    4550 scope.go:122] "RemoveContainer" containerID="5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83"
Nov 03 01:13:06.857277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.857252    4550 scope.go:122] "RemoveContainer" containerID="5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83"
Nov 03 01:13:06.857664 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:06.857641    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83\": container with ID starting with 5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83 not found: ID does not exist" containerID="5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83"
Nov 03 01:13:06.857760 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.857677    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83"} err="failed to get container status \"5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83\": rpc error: code = NotFound desc = could not find container \"5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83\": container with ID starting with 5cf3fac99ec75d2288d026cdf93c511681dc0bf646d5e5dc8ccb515620468d83 not found: ID does not exist"
Nov 03 01:13:06.884601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884566    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache\") pod \"a20455af-f5bf-41ca-a942-8c2b6b35d962\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") "
Nov 03 01:13:06.884795 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884663    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm\") pod \"a20455af-f5bf-41ca-a942-8c2b6b35d962\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") "
Nov 03 01:13:06.884795 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884717    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home\") pod \"776a620a-09d1-44e1-8953-d31795378672\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") "
Nov 03 01:13:06.884795 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884768    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir\") pod \"776a620a-09d1-44e1-8953-d31795378672\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") "
Nov 03 01:13:06.884982 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884819    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir\") pod \"a20455af-f5bf-41ca-a942-8c2b6b35d962\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") "
Nov 03 01:13:06.884982 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884910    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs\" (UniqueName: \"kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs\") pod \"a20455af-f5bf-41ca-a942-8c2b6b35d962\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") "
Nov 03 01:13:06.884982 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884967    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm\") pod \"776a620a-09d1-44e1-8953-d31795378672\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") "
Nov 03 01:13:06.885146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.884998    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v\" (UniqueName: \"kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v\") pod \"a20455af-f5bf-41ca-a942-8c2b6b35d962\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") "
Nov 03 01:13:06.885146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.885034    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9\" (UniqueName: \"kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9\") pod \"776a620a-09d1-44e1-8953-d31795378672\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") "
Nov 03 01:13:06.885146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.885062    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs\" (UniqueName: \"kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs\") pod \"776a620a-09d1-44e1-8953-d31795378672\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") "
Nov 03 01:13:06.885320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.885151    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home\") pod \"a20455af-f5bf-41ca-a942-8c2b6b35d962\" (UID: \"a20455af-f5bf-41ca-a942-8c2b6b35d962\") "
Nov 03 01:13:06.885320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.885183    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache\") pod \"776a620a-09d1-44e1-8953-d31795378672\" (UID: \"776a620a-09d1-44e1-8953-d31795378672\") "
Nov 03 01:13:06.889741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.887012    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home" pod "776a620a-09d1-44e1-8953-d31795378672" (UID: "776a620a-09d1-44e1-8953-d31795378672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.889741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.887768    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir" pod "776a620a-09d1-44e1-8953-d31795378672" (UID: "776a620a-09d1-44e1-8953-d31795378672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.889741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.888128    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache" pod "776a620a-09d1-44e1-8953-d31795378672" (UID: "776a620a-09d1-44e1-8953-d31795378672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.891308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.890943    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs" pod "776a620a-09d1-44e1-8953-d31795378672" (UID: "776a620a-09d1-44e1-8953-d31795378672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:06.891308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.891235    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir" pod "a20455af-f5bf-41ca-a942-8c2b6b35d962" (UID: "a20455af-f5bf-41ca-a942-8c2b6b35d962"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.891308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.891286    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home" pod "a20455af-f5bf-41ca-a942-8c2b6b35d962" (UID: "a20455af-f5bf-41ca-a942-8c2b6b35d962"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.891582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.891312    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs" pod "a20455af-f5bf-41ca-a942-8c2b6b35d962" (UID: "a20455af-f5bf-41ca-a942-8c2b6b35d962"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:06.891582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.891377    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm" pod "776a620a-09d1-44e1-8953-d31795378672" (UID: "776a620a-09d1-44e1-8953-d31795378672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.892102 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.892082    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache" pod "a20455af-f5bf-41ca-a942-8c2b6b35d962" (UID: "a20455af-f5bf-41ca-a942-8c2b6b35d962"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.893285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.893258    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9" pod "776a620a-09d1-44e1-8953-d31795378672" (UID: "776a620a-09d1-44e1-8953-d31795378672"). InnerVolumeSpecName "kube-api-access-fd5m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:06.893775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.893740    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v" pod "a20455af-f5bf-41ca-a942-8c2b6b35d962" (UID: "a20455af-f5bf-41ca-a942-8c2b6b35d962"). InnerVolumeSpecName "kube-api-access-8w48v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:06.895932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.895899    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm" pod "a20455af-f5bf-41ca-a942-8c2b6b35d962" (UID: "a20455af-f5bf-41ca-a942-8c2b6b35d962"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.987135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987089    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987131    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987147    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987161    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a20455af-f5bf-41ca-a942-8c2b6b35d962-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987172    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987183    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w48v\" (UniqueName: \"kubernetes.io/projected/a20455af-f5bf-41ca-a942-8c2b6b35d962-kube-api-access-8w48v\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987193    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5m9\" (UniqueName: \"kubernetes.io/projected/776a620a-09d1-44e1-8953-d31795378672-kube-api-access-fd5m9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987202    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776a620a-09d1-44e1-8953-d31795378672-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987210    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987219    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776a620a-09d1-44e1-8953-d31795378672-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987227    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.987564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:06.987235    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a20455af-f5bf-41ca-a942-8c2b6b35d962-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.177071 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.177033    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"]
Nov 03 01:13:07.181472 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.181446    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-2-w0-fc9316-kserve-7699b8448b-hhbwj"]
Nov 03 01:13:07.213645 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.213612    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"]
Nov 03 01:13:07.216270 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.216249    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/simulator-w0-fc9316-kserve-dbc4bf9d7-9z7pb"]
Nov 03 01:13:07.571608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.571547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:07.672699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.672665    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:07.808099 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.808065    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776a620a-09d1-44e1-8953-d31795378672" path="/var/lib/kubelet/pods/776a620a-09d1-44e1-8953-d31795378672/volumes"
Nov 03 01:13:07.808529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.808515    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" path="/var/lib/kubelet/pods/a20455af-f5bf-41ca-a942-8c2b6b35d962/volumes"
Nov 03 01:13:07.819476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:07.819449    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:09.462142 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:09.462108    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:10.431456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:10.431071    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:11.123625 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:11.123587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:12.028129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.028095    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"]
Nov 03 01:13:12.041175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.041147    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a6c39b0-8xns6"]
Nov 03 01:13:12.725276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.725240    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:12.809542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.808655    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"]
Nov 03 01:13:12.812908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.812874    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ee38c88b-5480-49c0-a796-d4c37df8e133" containerName="revoke-keys"
Nov 03 01:13:12.812908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.812907    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee38c88b-5480-49c0-a796-d4c37df8e133" containerName="revoke-keys"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.812925    4550 cpu_manager.go:414] "containerMap: removing container" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.812934    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.812969    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.812979    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.813139    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee38c88b-5480-49c0-a796-d4c37df8e133" containerName="revoke-keys"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.813153    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="776a620a-09d1-44e1-8953-d31795378672" containerName="main"
Nov 03 01:13:12.813175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.813170    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a20455af-f5bf-41ca-a942-8c2b6b35d962" containerName="main"
Nov 03 01:13:12.818853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.818830    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.838843 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.838813    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"]
Nov 03 01:13:12.847704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.847669    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5c109b-openshift-default-dockercfg-lfstj\""
Nov 03 01:13:12.862184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862156    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862199    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862231    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bll4z\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862356    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862391    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862422    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862560    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862600    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.862723 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.862643    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.963851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.963814    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.963876    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.963913    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bll4z\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964160    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964206    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964253    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964304    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964331    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964336    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964395    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964464    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964607    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.964897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.964627    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.965268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.965241    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.967344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.967308    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.967683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.967659    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.983997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.983930    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:12.984552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:12.984529    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bll4z\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:13.133503 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.133466    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:13.449699 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:13:13.449667    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424b73b2_0601_44b7_8aa2_20734d0aa15a.slice/crio-d4e38ac8ae30d1dce81a026624142cf9263586698e9ea395ecf6b23f241b93c1 WatchSource:0}: Error finding container d4e38ac8ae30d1dce81a026624142cf9263586698e9ea395ecf6b23f241b93c1: Status 404 returned error can't find the container with id d4e38ac8ae30d1dce81a026624142cf9263586698e9ea395ecf6b23f241b93c1
Nov 03 01:13:13.449835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.449795    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"]
Nov 03 01:13:13.452219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.452188    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:13:13.452336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.452260    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:13:13.452336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.452306    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:13:13.814206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.814110    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af6a7b0-b2eb-490b-b888-3453657ed551" path="/var/lib/kubelet/pods/0af6a7b0-b2eb-490b-b888-3453657ed551/volumes"
Nov 03 01:13:13.891068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.891019    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" event={"ID":"424b73b2-0601-44b7-8aa2-20734d0aa15a","Type":"ContainerStarted","Data":"a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b"}
Nov 03 01:13:13.891068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.891121    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" event={"ID":"424b73b2-0601-44b7-8aa2-20734d0aa15a","Type":"ContainerStarted","Data":"d4e38ac8ae30d1dce81a026624142cf9263586698e9ea395ecf6b23f241b93c1"}
Nov 03 01:13:13.938885 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:13.938836    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podStartSLOduration=1.938822372 podStartE2EDuration="1.938822372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:13.935823391 +0000 UTC m=+4076.257360191" watchObservedRunningTime="2026-11-03 01:13:13.938822372 +0000 UTC m=+4076.260359172"
Nov 03 01:13:14.136567 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:14.136424    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:14.138956 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:14.138915    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:13:14.139137 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:14.138988    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 01:13:14.334884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:14.334848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:15.134736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:15.134697    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:13:15.135165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:15.134752    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 01:13:15.723068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:15.722763    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:15.823633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:15.823595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:16.134547 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:16.134422    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:13:16.134547 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:16.134513    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 01:13:16.325112 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:16.325078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:16.438089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:16.438054    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:17.134155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:17.134115    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:13:17.134524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:17.134179    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 01:13:18.120769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:18.120734    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:18.134826 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:18.134793    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:13:18.135006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:18.134844    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 01:13:19.134563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:19.134522    4550 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Nov 03 01:13:19.134980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:19.134594    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Nov 03 01:13:19.708287 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:19.708250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:20.169691 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:20.169653    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:20.170197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:20.170164    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:20.171777 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:20.171750    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:13:21.373711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:21.373674    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:22.889041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:22.889010    4550 scope.go:122] "RemoveContainer" containerID="4b28c045b75826c31a6eb40f948d7c917e4ac76b22118c940340264ebbab4453"
Nov 03 01:13:22.901135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:22.901108    4550 scope.go:122] "RemoveContainer" containerID="94474e31501912db7151a7e8cc1de32c138b00fd156f84d2d8b58bf9fde959d8"
Nov 03 01:13:22.909897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:22.909874    4550 scope.go:122] "RemoveContainer" containerID="29f849d9b33a5ce1095fa2492af2d6425cfbc9dc2a6511d2bfb4978207269939"
Nov 03 01:13:22.920136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:22.920111    4550 scope.go:122] "RemoveContainer" containerID="43ce1673ff2129c43a3052d70aa593c39780d22383110c88542341bcb26be8f9"
Nov 03 01:13:22.929239 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:22.929218    4550 scope.go:122] "RemoveContainer" containerID="f6d4ec65d3f3d4575619ef1e5b1ebc3879fc0abd03addbaf4c19696fd18d7e59"
Nov 03 01:13:22.949347 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:22.949302    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:24.439298 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:24.439263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:24.539247 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:24.539211    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:24.640655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:24.640619    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:24.725887 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:24.725794    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:13:25.165540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:25.165492    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:25.536151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:25.536063    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:26.298089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:26.298051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:26.495610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:26.495572    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:28.234423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:28.234388    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:28.678197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:28.678159    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:28.781423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:28.781386    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:29.426842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:29.426742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:30.061683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:30.061642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:31.817789 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:31.817754    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:32.387116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:32.387079    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"]
Nov 03 01:13:32.421136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:32.421105    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"]
Nov 03 01:13:32.535555 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:32.535517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:32.640906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:32.640810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:33.428357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:33.428326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:33.728609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:33.728530    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:35.347993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:35.347959    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:35.449507 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:35.449470    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:35.549527 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:35.549493    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:37.335186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:37.335147    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:37.387675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:37.387638    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerName="payload-processing" containerID="cri-o://a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5" gracePeriod=25
Nov 03 01:13:37.422920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:37.422385    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerName="payload-pre-processing" containerID="cri-o://98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7" gracePeriod=25
Nov 03 01:13:39.045257 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:39.045219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:39.145328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:39.145276    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:40.116030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:40.115989    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:40.116454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:40.115991    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:40.907289 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:40.907247    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:42.643534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:42.643500    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:43.995951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:43.995917    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:44.620340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:44.620299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:46.226462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:46.226398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:47.834019 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:47.833985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:49.244551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:49.244511    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:49.942396 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:49.942359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:50.116514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:50.116474    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:50.116693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:50.116485    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:13:50.486483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:50.486425    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:50.600306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:50.600273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:51.221990 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:51.221960    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:51.589815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:51.589726    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"]
Nov 03 01:13:51.590383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:51.589977    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy" containerID="cri-o://0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f" gracePeriod=30
Nov 03 01:13:51.619648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:51.619615    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:51.717865 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:51.717834    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:53.446960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:53.446922    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:53.546145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:53.546107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:54.206866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:54.206827    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:54.588391 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:54.588300    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"]
Nov 03 01:13:54.588919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:54.588602    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" containerID="cri-o://2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6" gracePeriod=30
Nov 03 01:13:54.605250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:54.605222    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:54.788617 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:54.788582    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:13:55.349905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:55.349869    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:55.450350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:55.450296    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:56.863824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.863798    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 01:13:56.965226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965195    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965253    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965327    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965356    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965403    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965464    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965514    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965540    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.965721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965587    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.965721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965701    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token\") pod \"f147a48b-fba0-4781-a730-9af999130396\" (UID: \"f147a48b-fba0-4781-a730-9af999130396\") "
Nov 03 01:13:56.966068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.965828    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.966068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966044    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.966178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966132    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.966412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966285    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.966528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966305    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.966528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966496    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.966528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966501    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:56.966528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.966512    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.968150 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.968122    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.968487 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.968458    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:56.968811 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.968792    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "kube-api-access-pv5x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:56.969105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.969075    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo" pod "f147a48b-fba0-4781-a730-9af999130396" (UID: "f147a48b-fba0-4781-a730-9af999130396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:13:56.982367 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.982337    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:13:56.982515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:56.982400    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:13:57.068003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.067972    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f147a48b-fba0-4781-a730-9af999130396-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.068003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.067999    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f147a48b-fba0-4781-a730-9af999130396-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.068003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.068009    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5x5\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-kube-api-access-pv5x5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.068325 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.068022    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f147a48b-fba0-4781-a730-9af999130396-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.068325 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.068031    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f147a48b-fba0-4781-a730-9af999130396-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.106051 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.106018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:57.209592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.209555    4550 generic.go:356] "Generic (PLEG): container finished" podID="f147a48b-fba0-4781-a730-9af999130396" containerID="0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f" exitCode=0
Nov 03 01:13:57.209746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.209605    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" event={"ID":"f147a48b-fba0-4781-a730-9af999130396","Type":"ContainerDied","Data":"0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f"}
Nov 03 01:13:57.209746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.209617    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"
Nov 03 01:13:57.209746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.209647    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv" event={"ID":"f147a48b-fba0-4781-a730-9af999130396","Type":"ContainerDied","Data":"47431d7a3557e32ca3efdbe135ecfd1f21326e140df3a8ed4967ba190aaef58f"}
Nov 03 01:13:57.209746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.209668    4550 scope.go:122] "RemoveContainer" containerID="0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f"
Nov 03 01:13:57.221169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.221148    4550 scope.go:122] "RemoveContainer" containerID="0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f"
Nov 03 01:13:57.221460 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:57.221441    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f\": container with ID starting with 0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f not found: ID does not exist" containerID="0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f"
Nov 03 01:13:57.221518 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.221470    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f"} err="failed to get container status \"0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f\": rpc error: code = NotFound desc = could not find container \"0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f\": container with ID starting with 0d5a1e4a56de923eb5f9fb563d03229ff4214f1e5fbb9064d5fd64093c787f1f not found: ID does not exist"
Nov 03 01:13:57.265299 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.265265    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"]
Nov 03 01:13:57.269008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.268983    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-clnjv"]
Nov 03 01:13:57.549278 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.549252    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 01:13:57.573804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.573774    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz\" (UniqueName: \"kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz\") pod \"493bc4be-10d4-40cc-baf8-04ce4cf49566\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") "
Nov 03 01:13:57.574003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.573963    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume\") pod \"493bc4be-10d4-40cc-baf8-04ce4cf49566\" (UID: \"493bc4be-10d4-40cc-baf8-04ce4cf49566\") "
Nov 03 01:13:57.574397 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.574365    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume" pod "493bc4be-10d4-40cc-baf8-04ce4cf49566" (UID: "493bc4be-10d4-40cc-baf8-04ce4cf49566"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:57.577247 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.576206    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz" pod "493bc4be-10d4-40cc-baf8-04ce4cf49566" (UID: "493bc4be-10d4-40cc-baf8-04ce4cf49566"). InnerVolumeSpecName "kube-api-access-mptgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:57.675876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.675842    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mptgz\" (UniqueName: \"kubernetes.io/projected/493bc4be-10d4-40cc-baf8-04ce4cf49566-kube-api-access-mptgz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.675876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.675871    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/493bc4be-10d4-40cc-baf8-04ce4cf49566-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.726291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.726232    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 01:13:57.776999 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.776968    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96\" (UniqueName: \"kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96\") pod \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") "
Nov 03 01:13:57.777218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.777203    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume\") pod \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\" (UID: \"0c231b8b-a71b-4c4b-9987-b45d6326a61e\") "
Nov 03 01:13:57.777695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.777652    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume" pod "0c231b8b-a71b-4c4b-9987-b45d6326a61e" (UID: "0c231b8b-a71b-4c4b-9987-b45d6326a61e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:57.779715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.779683    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96" pod "0c231b8b-a71b-4c4b-9987-b45d6326a61e" (UID: "0c231b8b-a71b-4c4b-9987-b45d6326a61e"). InnerVolumeSpecName "kube-api-access-s4b96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:57.809979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.809946    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f147a48b-fba0-4781-a730-9af999130396" path="/var/lib/kubelet/pods/f147a48b-fba0-4781-a730-9af999130396/volumes"
Nov 03 01:13:57.878652 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.878618    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c231b8b-a71b-4c4b-9987-b45d6326a61e-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.878652 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:57.878651    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4b96\" (UniqueName: \"kubernetes.io/projected/0c231b8b-a71b-4c4b-9987-b45d6326a61e-kube-api-access-s4b96\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:58.218043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.218005    4550 generic.go:356] "Generic (PLEG): container finished" podID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerID="98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7" exitCode=0
Nov 03 01:13:58.218269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.218051    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"
Nov 03 01:13:58.218269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.218085    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" event={"ID":"0c231b8b-a71b-4c4b-9987-b45d6326a61e","Type":"ContainerDied","Data":"98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7"}
Nov 03 01:13:58.218269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.218141    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw" event={"ID":"0c231b8b-a71b-4c4b-9987-b45d6326a61e","Type":"ContainerDied","Data":"f91bdae5baaf0749c55ac703421e6dada87c633e11b6794c8d4c5891fef17284"}
Nov 03 01:13:58.218269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.218154    4550 scope.go:122] "RemoveContainer" containerID="98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7"
Nov 03 01:13:58.219727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.219697    4550 generic.go:356] "Generic (PLEG): container finished" podID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerID="a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5" exitCode=0
Nov 03 01:13:58.219824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.219731    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" event={"ID":"493bc4be-10d4-40cc-baf8-04ce4cf49566","Type":"ContainerDied","Data":"a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5"}
Nov 03 01:13:58.219824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.219761    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"
Nov 03 01:13:58.219915 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.219765    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb" event={"ID":"493bc4be-10d4-40cc-baf8-04ce4cf49566","Type":"ContainerDied","Data":"aa4cb5204b46ef54894ac956d2e6c48db6b7d5db7805868364358c083b56a0bf"}
Nov 03 01:13:58.230319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.230272    4550 scope.go:122] "RemoveContainer" containerID="98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7"
Nov 03 01:13:58.230696 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:58.230675    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7\": container with ID starting with 98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7 not found: ID does not exist" containerID="98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7"
Nov 03 01:13:58.230778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.230705    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7"} err="failed to get container status \"98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7\": rpc error: code = NotFound desc = could not find container \"98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7\": container with ID starting with 98591b494ef57be0c0ee471d92b045f2dc5805fa059132dcf2b25d4879b5d2a7 not found: ID does not exist"
Nov 03 01:13:58.230778 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.230727    4550 scope.go:122] "RemoveContainer" containerID="a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5"
Nov 03 01:13:58.239383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.239350    4550 scope.go:122] "RemoveContainer" containerID="a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5"
Nov 03 01:13:58.239662 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:13:58.239644    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5\": container with ID starting with a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5 not found: ID does not exist" containerID="a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5"
Nov 03 01:13:58.239724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.239669    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5"} err="failed to get container status \"a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5\": rpc error: code = NotFound desc = could not find container \"a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5\": container with ID starting with a4de59df2ad5fd238fed91027359d9bb10a948cb7e93b90f6918311cede196e5 not found: ID does not exist"
Nov 03 01:13:58.271394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.271327    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"]
Nov 03 01:13:58.283903 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.283874    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdacdb-847666b4ff-wktsw"]
Nov 03 01:13:58.309918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.309882    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"]
Nov 03 01:13:58.312459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.312425    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdacdb-6dfcc5477c-7n7vb"]
Nov 03 01:13:58.786890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:58.786846    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:13:59.810929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.810896    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" path="/var/lib/kubelet/pods/0c231b8b-a71b-4c4b-9987-b45d6326a61e/volumes"
Nov 03 01:13:59.811350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.811329    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" path="/var/lib/kubelet/pods/493bc4be-10d4-40cc-baf8-04ce4cf49566/volumes"
Nov 03 01:13:59.869415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.869341    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 01:13:59.904813 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.904776    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.904823    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.904859    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.904879    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.904907    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905093    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905127    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:59.905237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905205    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905408 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905229    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:59.905408 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905276    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905408 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905321    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token\") pod \"ff25e55a-ca15-4609-afcc-656560b9bc51\" (UID: \"ff25e55a-ca15-4609-afcc-656560b9bc51\") "
Nov 03 01:13:59.905408 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905353    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:59.905662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905457    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:59.905721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.905699    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:59.906036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.906011    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:59.906141 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.906040    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff25e55a-ca15-4609-afcc-656560b9bc51-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:59.906141 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.906095    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:59.906141 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.906111    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:59.906141 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.906125    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:13:59.907610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.907581    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "kube-api-access-8k6rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:59.907748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.907712    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:59.907860 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.907845    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:59.908030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:13:59.908006    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo" pod "ff25e55a-ca15-4609-afcc-656560b9bc51" (UID: "ff25e55a-ca15-4609-afcc-656560b9bc51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:00.007643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.007602    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:00.007643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.007640    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:00.007643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.007650    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:00.008030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.007659    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k6rh\" (UniqueName: \"kubernetes.io/projected/ff25e55a-ca15-4609-afcc-656560b9bc51-kube-api-access-8k6rh\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:00.235933 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.235897    4550 generic.go:356] "Generic (PLEG): container finished" podID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerID="2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6" exitCode=0
Nov 03 01:14:00.236155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.235943    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"
Nov 03 01:14:00.236155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.235959    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" event={"ID":"ff25e55a-ca15-4609-afcc-656560b9bc51","Type":"ContainerDied","Data":"2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6"}
Nov 03 01:14:00.236155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.236009    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9" event={"ID":"ff25e55a-ca15-4609-afcc-656560b9bc51","Type":"ContainerDied","Data":"7a823876d79114ddc8f386d515603fe9e7d5ae95c4cf071c91e86404c4c505da"}
Nov 03 01:14:00.236155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.236033    4550 scope.go:122] "RemoveContainer" containerID="2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6"
Nov 03 01:14:00.246900 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.246874    4550 scope.go:122] "RemoveContainer" containerID="2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6"
Nov 03 01:14:00.247184 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:14:00.247165    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6\": container with ID starting with 2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6 not found: ID does not exist" containerID="2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6"
Nov 03 01:14:00.247240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.247192    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6"} err="failed to get container status \"2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6\": rpc error: code = NotFound desc = could not find container \"2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6\": container with ID starting with 2de5bfa0cbcd34afd525bd3bcf0f7bae4849172261c5590a1311e9eadf7457e6 not found: ID does not exist"
Nov 03 01:14:00.289425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.289394    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"]
Nov 03 01:14:00.302307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.302279    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-sqdz9"]
Nov 03 01:14:00.337579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:00.337547    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:01.344425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:01.344385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:01.442092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:01.442058    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:01.808156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:01.808122    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" path="/var/lib/kubelet/pods/ff25e55a-ca15-4609-afcc-656560b9bc51/volumes"
Nov 03 01:14:02.112044 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:02.111950    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:02.394108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:02.394006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:03.927814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:03.927777    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:05.627627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:05.627588    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:07.424578 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:07.424543    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:08.598909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:08.598861    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:09.111622 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:09.111592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:10.733401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:10.733359    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:12.347691 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:12.347651    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:14.016840 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:14.016803    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:15.404971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:15.404915    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:15.963937 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:15.963902    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:17.337145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:17.337107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:17.671486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:17.671422    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:19.439654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:19.439616    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:21.083335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:21.083301    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:22.722625 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:22.722587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:22.992646 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:22.992557    4550 scope.go:122] "RemoveContainer" containerID="31410ddd7d4d2e663a11429f55f325241dd4d6e817bf118ef69bb3b17f0d287e"
Nov 03 01:14:23.066650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:23.064389    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:23.164580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:23.164545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:23.865377 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:23.865335    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:24.235869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:24.235837    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:24.652220 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:24.652190    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:24.739996 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:14:24.739957    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:14:25.025234 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:25.025148    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:26.341272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:26.341238    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:27.757352 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757316    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"]
Nov 03 01:14:27.757875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757807    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerName="payload-pre-processing"
Nov 03 01:14:27.757875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757820    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerName="payload-pre-processing"
Nov 03 01:14:27.757875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757834    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy"
Nov 03 01:14:27.757875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757839    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy"
Nov 03 01:14:27.757875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757850    4550 cpu_manager.go:414] "containerMap: removing container" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerName="payload-processing"
Nov 03 01:14:27.757875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757855    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerName="payload-processing"
Nov 03 01:14:27.758224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757885    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy"
Nov 03 01:14:27.758224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757890    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy"
Nov 03 01:14:27.758224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757961    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f147a48b-fba0-4781-a730-9af999130396" containerName="istio-proxy"
Nov 03 01:14:27.758224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757969    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c231b8b-a71b-4c4b-9987-b45d6326a61e" containerName="payload-pre-processing"
Nov 03 01:14:27.758224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757975    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff25e55a-ca15-4609-afcc-656560b9bc51" containerName="istio-proxy"
Nov 03 01:14:27.758224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.757982    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="493bc4be-10d4-40cc-baf8-04ce4cf49566" containerName="payload-processing"
Nov 03 01:14:27.761193 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.761173    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.779157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.779135    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"]
Nov 03 01:14:27.788022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.787999    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:14:27.788460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.788424    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:14:27.868556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.868522    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.868787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.868764    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq4p6\" (UniqueName: \"kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.868919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.868894    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.970373 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.970316    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq4p6\" (UniqueName: \"kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.970599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.970392    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.970599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.970536    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.971161 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.971139    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.972692 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.972670    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.989738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.989713    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq4p6\" (UniqueName: \"kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6\") pod \"maas-api-revoke-sub-022d7e0c-cz77s\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:27.994732 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:27.994703    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:28.076073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:28.075980    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:28.374543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:28.374501    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"]
Nov 03 01:14:28.374785 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:14:28.374761    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583e0c31_6e9f_44a0_b4d0_3a33f9d84d2f.slice/crio-9b738495b4d7b4728a45c3f7a4d88c71823467bf29acb2165d9ed6763b2c4086 WatchSource:0}: Error finding container 9b738495b4d7b4728a45c3f7a4d88c71823467bf29acb2165d9ed6763b2c4086: Status 404 returned error can't find the container with id 9b738495b4d7b4728a45c3f7a4d88c71823467bf29acb2165d9ed6763b2c4086
Nov 03 01:14:28.432221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:28.432193    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s" event={"ID":"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f","Type":"ContainerStarted","Data":"9b738495b4d7b4728a45c3f7a4d88c71823467bf29acb2165d9ed6763b2c4086"}
Nov 03 01:14:29.439170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:29.439073    4550 generic.go:356] "Generic (PLEG): container finished" podID="583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" containerID="63e024f2ed3fdf0a0c71c788cf2890f073897d9f5e9fb736a300cf11e9f3694e" exitCode=0
Nov 03 01:14:29.439632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:29.439162    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s" event={"ID":"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f","Type":"ContainerDied","Data":"63e024f2ed3fdf0a0c71c788cf2890f073897d9f5e9fb736a300cf11e9f3694e"}
Nov 03 01:14:29.678995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:29.678958    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:30.586946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.586923    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:30.705580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.705495    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6\" (UniqueName: \"kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6\") pod \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") "
Nov 03 01:14:30.705743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.705688    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca\") pod \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") "
Nov 03 01:14:30.705794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.705741    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca\") pod \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\" (UID: \"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f\") "
Nov 03 01:14:30.706525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.706494    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca" pod "583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" (UID: "583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:30.707788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.707749    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca" pod "583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" (UID: "583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:30.708060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.708043    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6" pod "583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" (UID: "583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f"). InnerVolumeSpecName "kube-api-access-pq4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:30.732363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.732333    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"]
Nov 03 01:14:30.807033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.807000    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.807033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.807032    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.807249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.807049    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq4p6\" (UniqueName: \"kubernetes.io/projected/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f-kube-api-access-pq4p6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.845260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.845227    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:30.951924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:30.951883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:31.451407 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:31.451375    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"
Nov 03 01:14:31.451606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:31.451411    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s" event={"ID":"583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f","Type":"ContainerDied","Data":"9b738495b4d7b4728a45c3f7a4d88c71823467bf29acb2165d9ed6763b2c4086"}
Nov 03 01:14:31.451606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:31.451489    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b738495b4d7b4728a45c3f7a4d88c71823467bf29acb2165d9ed6763b2c4086"
Nov 03 01:14:31.534110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:31.534068    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:31.635802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:31.635762    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:33.330816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:33.330780    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:35.023908 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:35.023868    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:35.733379 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:35.733294    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerName="payload-pre-processing" containerID="cri-o://06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808" gracePeriod=25
Nov 03 01:14:36.663696 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:36.663659    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:37.478886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:37.478855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:38.100934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.100891    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"]
Nov 03 01:14:38.101710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.101691    4550 cpu_manager.go:414] "containerMap: removing container" podUID="583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" containerName="revoke-keys"
Nov 03 01:14:38.101767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.101714    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" containerName="revoke-keys"
Nov 03 01:14:38.101828 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.101816    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" containerName="revoke-keys"
Nov 03 01:14:38.105686 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.105669    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.121357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.121330    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"]
Nov 03 01:14:38.131073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.131049    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:14:38.131217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.131066    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:14:38.194679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.194647    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.194867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.194791    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.194933 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.194871    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5d9\" (UniqueName: \"kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.224324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.224293    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:38.297340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.296235    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.297340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.296413    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.297340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.296519    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5d9\" (UniqueName: \"kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.297779 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.297720    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.301621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.301596    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.316404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.316380    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5d9\" (UniqueName: \"kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9\") pod \"maas-api-revoke-sub-2f7606b7-f4lcm\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.418383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.418356    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:38.716618 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:14:38.716586    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f61bd1_aef2_4709_aa1d_d0e2d78614e1.slice/crio-2817aac15a158b6f31a59e535b9236f6cd8eba1e31248e0a277d5f374817e782 WatchSource:0}: Error finding container 2817aac15a158b6f31a59e535b9236f6cd8eba1e31248e0a277d5f374817e782: Status 404 returned error can't find the container with id 2817aac15a158b6f31a59e535b9236f6cd8eba1e31248e0a277d5f374817e782
Nov 03 01:14:38.717184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:38.717156    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"]
Nov 03 01:14:39.507551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:39.507520    4550 generic.go:356] "Generic (PLEG): container finished" podID="f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" containerID="74702cd8a2498a34b98b72a729deaaf5de682643a60e84eaf69bd360d0605af5" exitCode=0
Nov 03 01:14:39.507916 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:39.507607    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm" event={"ID":"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1","Type":"ContainerDied","Data":"74702cd8a2498a34b98b72a729deaaf5de682643a60e84eaf69bd360d0605af5"}
Nov 03 01:14:39.507916 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:39.507673    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm" event={"ID":"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1","Type":"ContainerStarted","Data":"2817aac15a158b6f31a59e535b9236f6cd8eba1e31248e0a277d5f374817e782"}
Nov 03 01:14:39.822055 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:39.821966    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:40.665279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.665252    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:40.833008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.832906    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca\") pod \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") "
Nov 03 01:14:40.833008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.832971    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca\") pod \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") "
Nov 03 01:14:40.833277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.833063    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9\" (UniqueName: \"kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9\") pod \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\" (UID: \"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1\") "
Nov 03 01:14:40.834690 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.834661    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca" pod "f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" (UID: "f6f61bd1-aef2-4709-aa1d-d0e2d78614e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:40.839249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.837126    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9" pod "f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" (UID: "f6f61bd1-aef2-4709-aa1d-d0e2d78614e1"). InnerVolumeSpecName "kube-api-access-mx5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:40.839249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.837174    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca" pod "f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" (UID: "f6f61bd1-aef2-4709-aa1d-d0e2d78614e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:40.939766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.939733    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:40.953337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.953301    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5d9\" (UniqueName: \"kubernetes.io/projected/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-kube-api-access-mx5d9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:40.953337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:40.953340    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:41.522069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:41.522040    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"
Nov 03 01:14:41.522069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:41.522057    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm" event={"ID":"f6f61bd1-aef2-4709-aa1d-d0e2d78614e1","Type":"ContainerDied","Data":"2817aac15a158b6f31a59e535b9236f6cd8eba1e31248e0a277d5f374817e782"}
Nov 03 01:14:41.522320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:41.522111    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2817aac15a158b6f31a59e535b9236f6cd8eba1e31248e0a277d5f374817e782"
Nov 03 01:14:41.696158 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:41.696122    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:42.039269 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:42.039229    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:43.338887 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:43.338848    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:44.236818 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:44.236787    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:44.936717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:44.936684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:46.513277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:46.513242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:47.897642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:47.897613    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:48.498934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:48.498883    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:49.108017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.107986    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"]
Nov 03 01:14:49.108504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.108489    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" containerName="revoke-keys"
Nov 03 01:14:49.108561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.108504    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" containerName="revoke-keys"
Nov 03 01:14:49.108599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.108583    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" containerName="revoke-keys"
Nov 03 01:14:49.111866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.111847    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.132113 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.132083    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"]
Nov 03 01:14:49.162105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.162078    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.162251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.162146    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6sc\" (UniqueName: \"kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.162300 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.162253    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.189076 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.189045    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:14:49.189213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.189193    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:14:49.222882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.222854    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"]
Nov 03 01:14:49.226676 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.226660    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.244200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.244175    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"]
Nov 03 01:14:49.263065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263039    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.263198 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263098    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.263276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263246    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.263338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263291    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6qw\" (UniqueName: \"kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.263338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263314    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6sc\" (UniqueName: \"kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.263572 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263400    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.263979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.263938    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.265678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.265636    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.284277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.284246    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6sc\" (UniqueName: \"kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc\") pod \"maas-api-revoke-sub-8b949a7a-xhdgg\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.364928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.364862    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.365060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.364957    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.365060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.365027    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6qw\" (UniqueName: \"kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.365494 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.365472    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.367273 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.367247    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.384674 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.384650    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6qw\" (UniqueName: \"kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw\") pod \"maas-api-revoke-sub-f58a3b7c-wq4nx\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.429969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.429942    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:49.540078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.540048    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:49.728550 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:14:49.728517    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9821e4_845e_4b4a_91f0_e067bdb9d5ce.slice/crio-f735b4416517aabf069d768d55d03dc42b28353a9e78f70c572f2985738eda18 WatchSource:0}: Error finding container f735b4416517aabf069d768d55d03dc42b28353a9e78f70c572f2985738eda18: Status 404 returned error can't find the container with id f735b4416517aabf069d768d55d03dc42b28353a9e78f70c572f2985738eda18
Nov 03 01:14:49.729810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.729783    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"]
Nov 03 01:14:49.838241 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:14:49.838207    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac5ef82_a949_4911_93e0_b3c5a84714b2.slice/crio-34de0f0e5b3958279d2036607cbfaf429c03f3093466a0c8d2dbcb2bd75439aa WatchSource:0}: Error finding container 34de0f0e5b3958279d2036607cbfaf429c03f3093466a0c8d2dbcb2bd75439aa: Status 404 returned error can't find the container with id 34de0f0e5b3958279d2036607cbfaf429c03f3093466a0c8d2dbcb2bd75439aa
Nov 03 01:14:49.838993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:49.838972    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"]
Nov 03 01:14:50.191322 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.191291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:50.606176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.606126    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx" event={"ID":"8ac5ef82-a949-4911-93e0-b3c5a84714b2","Type":"ContainerStarted","Data":"31c008f3511fd6eb7699e33be992887fd143080e714847b812d12929a6857886"}
Nov 03 01:14:50.606394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.606201    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx" event={"ID":"8ac5ef82-a949-4911-93e0-b3c5a84714b2","Type":"ContainerStarted","Data":"34de0f0e5b3958279d2036607cbfaf429c03f3093466a0c8d2dbcb2bd75439aa"}
Nov 03 01:14:50.608037 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.608008    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg" event={"ID":"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce","Type":"ContainerStarted","Data":"8c454f76c6eefbdbeb6170b0c451c4960929586287d387aad0b8958f1eb592d6"}
Nov 03 01:14:50.608162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.608058    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg" event={"ID":"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce","Type":"ContainerStarted","Data":"f735b4416517aabf069d768d55d03dc42b28353a9e78f70c572f2985738eda18"}
Nov 03 01:14:50.646766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.646706    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx" podStartSLOduration=1.64669099 podStartE2EDuration="1.64669099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:50.645481264 +0000 UTC m=+4172.967018065" watchObservedRunningTime="2026-11-03 01:14:50.64669099 +0000 UTC m=+4172.968227803"
Nov 03 01:14:50.687875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:50.687779    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg" podStartSLOduration=1.68776095 podStartE2EDuration="1.68776095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:50.685394717 +0000 UTC m=+4173.006931543" watchObservedRunningTime="2026-11-03 01:14:50.68776095 +0000 UTC m=+4173.009297750"
Nov 03 01:14:51.614825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:51.614787    4550 generic.go:356] "Generic (PLEG): container finished" podID="8ac5ef82-a949-4911-93e0-b3c5a84714b2" containerID="31c008f3511fd6eb7699e33be992887fd143080e714847b812d12929a6857886" exitCode=0
Nov 03 01:14:51.615263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:51.614875    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx" event={"ID":"8ac5ef82-a949-4911-93e0-b3c5a84714b2","Type":"ContainerDied","Data":"31c008f3511fd6eb7699e33be992887fd143080e714847b812d12929a6857886"}
Nov 03 01:14:51.616412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:51.616386    4550 generic.go:356] "Generic (PLEG): container finished" podID="2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" containerID="8c454f76c6eefbdbeb6170b0c451c4960929586287d387aad0b8958f1eb592d6" exitCode=0
Nov 03 01:14:51.616561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:51.616465    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg" event={"ID":"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce","Type":"ContainerDied","Data":"8c454f76c6eefbdbeb6170b0c451c4960929586287d387aad0b8958f1eb592d6"}
Nov 03 01:14:52.027165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.027133    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:52.039493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.039464    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:52.778249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.778218    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:52.809329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.809309    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:52.905619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.905585    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc\" (UniqueName: \"kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc\") pod \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") "
Nov 03 01:14:52.905825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.905643    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca\") pod \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") "
Nov 03 01:14:52.905825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.905705    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca\") pod \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") "
Nov 03 01:14:52.905825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.905816    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca\") pod \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") "
Nov 03 01:14:52.906011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.905927    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca\") pod \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\" (UID: \"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce\") "
Nov 03 01:14:52.906011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.905985    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw\" (UniqueName: \"kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw\") pod \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\" (UID: \"8ac5ef82-a949-4911-93e0-b3c5a84714b2\") "
Nov 03 01:14:52.906144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.906116    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca" pod "8ac5ef82-a949-4911-93e0-b3c5a84714b2" (UID: "8ac5ef82-a949-4911-93e0-b3c5a84714b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:52.906323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.906300    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca" pod "2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" (UID: "2f9821e4-845e-4b4a-91f0-e067bdb9d5ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:52.906627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.906590    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:52.906627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.906624    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:52.908082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.908050    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca" pod "2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" (UID: "2f9821e4-845e-4b4a-91f0-e067bdb9d5ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:52.908213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.908089    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw" pod "8ac5ef82-a949-4911-93e0-b3c5a84714b2" (UID: "8ac5ef82-a949-4911-93e0-b3c5a84714b2"). InnerVolumeSpecName "kube-api-access-zh6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:52.908275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.908253    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca" pod "8ac5ef82-a949-4911-93e0-b3c5a84714b2" (UID: "8ac5ef82-a949-4911-93e0-b3c5a84714b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:52.908275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:52.908268    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc" pod "2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" (UID: "2f9821e4-845e-4b4a-91f0-e067bdb9d5ce"). InnerVolumeSpecName "kube-api-access-xf6sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:53.008275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.008245    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf6sc\" (UniqueName: \"kubernetes.io/projected/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-kube-api-access-xf6sc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:53.008275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.008273    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:53.008275 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.008282    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac5ef82-a949-4911-93e0-b3c5a84714b2-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:53.008564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.008290    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh6qw\" (UniqueName: \"kubernetes.io/projected/8ac5ef82-a949-4911-93e0-b3c5a84714b2-kube-api-access-zh6qw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:53.411772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.411742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:53.511516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.511481    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:53.629209 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.629182    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"
Nov 03 01:14:53.629398 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.629217    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg" event={"ID":"2f9821e4-845e-4b4a-91f0-e067bdb9d5ce","Type":"ContainerDied","Data":"f735b4416517aabf069d768d55d03dc42b28353a9e78f70c572f2985738eda18"}
Nov 03 01:14:53.629398 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.629265    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f735b4416517aabf069d768d55d03dc42b28353a9e78f70c572f2985738eda18"
Nov 03 01:14:53.630780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.630753    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx" event={"ID":"8ac5ef82-a949-4911-93e0-b3c5a84714b2","Type":"ContainerDied","Data":"34de0f0e5b3958279d2036607cbfaf429c03f3093466a0c8d2dbcb2bd75439aa"}
Nov 03 01:14:53.630895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.630781    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"
Nov 03 01:14:53.630895 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.630799    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34de0f0e5b3958279d2036607cbfaf429c03f3093466a0c8d2dbcb2bd75439aa"
Nov 03 01:14:53.874471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:53.874352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:54.149584 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:54.149491    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:54.530485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:54.530419    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:54.723109 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:14:54.723072    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:14:55.043535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.043503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:55.508780 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.508746    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"]
Nov 03 01:14:55.509212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.509198    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8ac5ef82-a949-4911-93e0-b3c5a84714b2" containerName="revoke-keys"
Nov 03 01:14:55.509212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.509212    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac5ef82-a949-4911-93e0-b3c5a84714b2" containerName="revoke-keys"
Nov 03 01:14:55.509305 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.509229    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" containerName="revoke-keys"
Nov 03 01:14:55.509305 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.509234    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" containerName="revoke-keys"
Nov 03 01:14:55.509371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.509323    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac5ef82-a949-4911-93e0-b3c5a84714b2" containerName="revoke-keys"
Nov 03 01:14:55.509371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.509333    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" containerName="revoke-keys"
Nov 03 01:14:55.512827 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.512809    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.531440 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.531402    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"]
Nov 03 01:14:55.540896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.540870    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:14:55.541040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.540978    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:14:55.641263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.641226    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrd7q\" (UniqueName: \"kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.641263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.641262    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.641524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.641487    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.743059 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.743029    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.743224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.743079    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrd7q\" (UniqueName: \"kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.743224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.743103    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.743659 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.743639    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.745471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.745446    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.781901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.781879    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrd7q\" (UniqueName: \"kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q\") pod \"maas-api-revoke-sub-01c73b89-bmnfd\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.826281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.826251    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:55.989350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:55.989323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:56.033248 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.030811    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 01:14:56.127019 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:14:56.126989    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c23b1de_e9f8_4c2f_89cb_cc658f4d5f65.slice/crio-e841f49ba4204b0629546d69281cf06b140f389417b2e7560a4e329d9cbd4fab WatchSource:0}: Error finding container e841f49ba4204b0629546d69281cf06b140f389417b2e7560a4e329d9cbd4fab: Status 404 returned error can't find the container with id e841f49ba4204b0629546d69281cf06b140f389417b2e7560a4e329d9cbd4fab
Nov 03 01:14:56.128465 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.128421    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"]
Nov 03 01:14:56.148411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.148390    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume\") pod \"dbe6388a-55ba-46d7-8c62-4f077991e60a\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") "
Nov 03 01:14:56.148533 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.148474    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r\" (UniqueName: \"kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r\") pod \"dbe6388a-55ba-46d7-8c62-4f077991e60a\" (UID: \"dbe6388a-55ba-46d7-8c62-4f077991e60a\") "
Nov 03 01:14:56.148721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.148698    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume" pod "dbe6388a-55ba-46d7-8c62-4f077991e60a" (UID: "dbe6388a-55ba-46d7-8c62-4f077991e60a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:56.148869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.148856    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe6388a-55ba-46d7-8c62-4f077991e60a-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:56.150420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.150374    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r" pod "dbe6388a-55ba-46d7-8c62-4f077991e60a" (UID: "dbe6388a-55ba-46d7-8c62-4f077991e60a"). InnerVolumeSpecName "kube-api-access-zvc9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:56.249707 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.249676    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvc9r\" (UniqueName: \"kubernetes.io/projected/dbe6388a-55ba-46d7-8c62-4f077991e60a-kube-api-access-zvc9r\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:56.672255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.672160    4550 generic.go:356] "Generic (PLEG): container finished" podID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerID="06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808" exitCode=0
Nov 03 01:14:56.672255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.672209    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"
Nov 03 01:14:56.672255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.672238    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" event={"ID":"dbe6388a-55ba-46d7-8c62-4f077991e60a","Type":"ContainerDied","Data":"06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808"}
Nov 03 01:14:56.672556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.672309    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76" event={"ID":"dbe6388a-55ba-46d7-8c62-4f077991e60a","Type":"ContainerDied","Data":"d3b2c73c73eb6411ad037d095cb32fd3380e4790808a8a13b61d42749842958b"}
Nov 03 01:14:56.672556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.672343    4550 scope.go:122] "RemoveContainer" containerID="06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808"
Nov 03 01:14:56.674010 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.673986    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd" event={"ID":"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65","Type":"ContainerStarted","Data":"279ac245be4752bdb3ae2b810e2e4a1454c6dc21332bff3b2ca6c8b5e13b0adf"}
Nov 03 01:14:56.674134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.674033    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd" event={"ID":"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65","Type":"ContainerStarted","Data":"e841f49ba4204b0629546d69281cf06b140f389417b2e7560a4e329d9cbd4fab"}
Nov 03 01:14:56.681974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.681955    4550 scope.go:122] "RemoveContainer" containerID="06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808"
Nov 03 01:14:56.682226 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:14:56.682207    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808\": container with ID starting with 06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808 not found: ID does not exist" containerID="06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808"
Nov 03 01:14:56.682279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.682235    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808"} err="failed to get container status \"06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808\": rpc error: code = NotFound desc = could not find container \"06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808\": container with ID starting with 06515639b5d5df39770bf17662c2f57b3a357839b72a698f82adee4e3c355808 not found: ID does not exist"
Nov 03 01:14:56.728950 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.728900    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd" podStartSLOduration=1.728888049 podStartE2EDuration="1.728888049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:56.727984026 +0000 UTC m=+4179.049520804" watchObservedRunningTime="2026-11-03 01:14:56.728888049 +0000 UTC m=+4179.050424902"
Nov 03 01:14:56.766593 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.766565    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"]
Nov 03 01:14:56.769823 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:56.769799    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-7ef1b6-55ccc79d7f-66p76"]
Nov 03 01:14:57.684174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:57.684137    4550 generic.go:356] "Generic (PLEG): container finished" podID="1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" containerID="279ac245be4752bdb3ae2b810e2e4a1454c6dc21332bff3b2ca6c8b5e13b0adf" exitCode=0
Nov 03 01:14:57.684631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:57.684221    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd" event={"ID":"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65","Type":"ContainerDied","Data":"279ac245be4752bdb3ae2b810e2e4a1454c6dc21332bff3b2ca6c8b5e13b0adf"}
Nov 03 01:14:57.714812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:57.714783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:57.810946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:57.810910    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" path="/var/lib/kubelet/pods/dbe6388a-55ba-46d7-8c62-4f077991e60a/volumes"
Nov 03 01:14:58.836637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.836611    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:58.982708 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.982628    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca\") pod \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") "
Nov 03 01:14:58.982869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.982752    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca\") pod \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") "
Nov 03 01:14:58.982869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.982804    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q\" (UniqueName: \"kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q\") pod \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\" (UID: \"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65\") "
Nov 03 01:14:58.983036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.983010    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca" pod "1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" (UID: "1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:58.983096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.983082    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:58.984825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.984796    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca" pod "1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" (UID: "1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:14:58.985047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:58.985023    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q" pod "1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" (UID: "1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65"). InnerVolumeSpecName "kube-api-access-xrd7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:59.084376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.084341    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:59.084376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.084365    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrd7q\" (UniqueName: \"kubernetes.io/projected/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65-kube-api-access-xrd7q\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:14:59.418501 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.418468    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:14:59.585692 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.585615    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"]
Nov 03 01:14:59.586329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.586313    4550 cpu_manager.go:414] "containerMap: removing container" podUID="1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" containerName="revoke-keys"
Nov 03 01:14:59.586382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.586332    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" containerName="revoke-keys"
Nov 03 01:14:59.586382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.586357    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerName="payload-pre-processing"
Nov 03 01:14:59.586382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.586365    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerName="payload-pre-processing"
Nov 03 01:14:59.586508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.586496    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" containerName="revoke-keys"
Nov 03 01:14:59.586551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.586511    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe6388a-55ba-46d7-8c62-4f077991e60a" containerName="payload-pre-processing"
Nov 03 01:14:59.590307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.590290    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.607748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.607719    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"]
Nov 03 01:14:59.646119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.646095    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"]
Nov 03 01:14:59.649743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.649727    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.670749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.670685    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"]
Nov 03 01:14:59.693108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.693069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6lf\" (UniqueName: \"kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.693249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.693132    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.693349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.693264    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.699363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.699341    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"
Nov 03 01:14:59.699620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.699352    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd" event={"ID":"1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65","Type":"ContainerDied","Data":"e841f49ba4204b0629546d69281cf06b140f389417b2e7560a4e329d9cbd4fab"}
Nov 03 01:14:59.699700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.699654    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e841f49ba4204b0629546d69281cf06b140f389417b2e7560a4e329d9cbd4fab"
Nov 03 01:14:59.794504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.794473    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6lf\" (UniqueName: \"kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.794653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.794514    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.794653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.794602    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm997\" (UniqueName: \"kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.794653 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.794630    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.794818 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.794700    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.794818 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.794738    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.795240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.795207    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.796994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.796972    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.813964 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.813938    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6lf\" (UniqueName: \"kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf\") pod \"maas-api-revoke-sub-33808623-l9fd5\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.895920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.895887    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.896384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.895951    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.896384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.896184    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm997\" (UniqueName: \"kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.896670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.896645    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.898914 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.898887    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.907883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.907859    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:14:59.919135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.919107    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm997\" (UniqueName: \"kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997\") pod \"maas-api-revoke-sub-c5375652-wrdgl\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.962578 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.962541    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:14:59.986833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:14:59.982673    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"]
Nov 03 01:15:00.031845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.013456    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"]
Nov 03 01:15:00.079877 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.079851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:00.146236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.146198    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"]
Nov 03 01:15:00.150929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.150903    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.159975 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.159947    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"]
Nov 03 01:15:00.165213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.165192    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.169484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.167918    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"]
Nov 03 01:15:00.172289 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.172270    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.173122 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.173098    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"]
Nov 03 01:15:00.178933 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.178912    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.203807 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.197649    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"]
Nov 03 01:15:00.206985 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.205665    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"]
Nov 03 01:15:00.211580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.211485    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"]
Nov 03 01:15:00.216118 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.214921    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"]
Nov 03 01:15:00.218655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.218631    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"]
Nov 03 01:15:00.223395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.222831    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.232648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.232626    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"]
Nov 03 01:15:00.240578 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.240009    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015206dc_1fd7_4f34_a0b0_89e0255b1720.slice/crio-52f5375e771b3cccb1abc3f6afed9fbe959a29a3baabdfc7420fcf9ab8005077 WatchSource:0}: Error finding container 52f5375e771b3cccb1abc3f6afed9fbe959a29a3baabdfc7420fcf9ab8005077: Status 404 returned error can't find the container with id 52f5375e771b3cccb1abc3f6afed9fbe959a29a3baabdfc7420fcf9ab8005077
Nov 03 01:15:00.242164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.241556    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"]
Nov 03 01:15:00.245796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.245381    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"]
Nov 03 01:15:00.252794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.250926    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.282166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.282140    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"]
Nov 03 01:15:00.311185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311146    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-9lk8s\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.311304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311206    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxng7\" (UniqueName: \"kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.311349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311295    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.311349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311330    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4ts\" (UniqueName: \"kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts\") pod \"maas-api-key-cleanup-29894475-9lk8s\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.311468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311407    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn8c\" (UniqueName: \"kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.311468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311424    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntg27\" (UniqueName: \"kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.311600 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311577    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vvv\" (UniqueName: \"kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.311673 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311658    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.311736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311692    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.311790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.311755    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.328709 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.328685    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8778b9c3_cee6_42d9_8ccd_ed5ddc189c84.slice/crio-6820ed87a6de772c7aeeadeb332279511125de57464954406dc814c24fc48be3 WatchSource:0}: Error finding container 6820ed87a6de772c7aeeadeb332279511125de57464954406dc814c24fc48be3: Status 404 returned error can't find the container with id 6820ed87a6de772c7aeeadeb332279511125de57464954406dc814c24fc48be3
Nov 03 01:15:00.330316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.330295    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"]
Nov 03 01:15:00.412395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412364    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4ts\" (UniqueName: \"kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts\") pod \"maas-api-key-cleanup-29894475-9lk8s\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.412585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412472    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn8c\" (UniqueName: \"kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.412585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412495    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntg27\" (UniqueName: \"kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.412585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412522    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vvv\" (UniqueName: \"kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.412585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412548    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.412585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412566    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvsp\" (UniqueName: \"kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.412585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412586    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.412978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412704    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.412978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412791    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.412978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412858    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-9lk8s\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.412978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.412935    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxng7\" (UniqueName: \"kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.413195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.413005    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.413416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.413392    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.413528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.413416    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.413528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.413397    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.413679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.413658    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-9lk8s\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.413739 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.413661    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.456606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.456536    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vvv\" (UniqueName: \"kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv\") pod \"maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.456606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.456538    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxng7\" (UniqueName: \"kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7\") pod \"maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.457140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.457113    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4ts\" (UniqueName: \"kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts\") pod \"maas-api-key-cleanup-29894475-9lk8s\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.457320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.457299    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntg27\" (UniqueName: \"kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.457813 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.457788    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn8c\" (UniqueName: \"kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.471318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.471296    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:00.484904 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.484882    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:00.498362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.498333    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:00.509986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.509815    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:00.515238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.514446    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvsp\" (UniqueName: \"kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.515238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.514550    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.515685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.515659    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.560633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.550216    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvsp\" (UniqueName: \"kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.560633 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.552849    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:00.581851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.581177    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:00.720209 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.720184    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5" event={"ID":"015206dc-1fd7-4f34-a0b0-89e0255b1720","Type":"ContainerStarted","Data":"52f5375e771b3cccb1abc3f6afed9fbe959a29a3baabdfc7420fcf9ab8005077"}
Nov 03 01:15:00.722150 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.722123    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl" event={"ID":"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84","Type":"ContainerStarted","Data":"dccd77d2983fac6b423a927381ce635d398668aa60f9651793563255e16b0de2"}
Nov 03 01:15:00.722264 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.722173    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl" event={"ID":"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84","Type":"ContainerStarted","Data":"6820ed87a6de772c7aeeadeb332279511125de57464954406dc814c24fc48be3"}
Nov 03 01:15:00.763468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.763327    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl" podStartSLOduration=1.763314182 podStartE2EDuration="1.763314182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:00.762221013 +0000 UTC m=+4183.083757833" watchObservedRunningTime="2026-11-03 01:15:00.763314182 +0000 UTC m=+4183.084850979"
Nov 03 01:15:00.799188 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.799154    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a1b0be_e2e6_43b7_bf30_77b8bb8b4a6f.slice/crio-50fb39d5fa4f7514890ec0bc9d55a9c8357d204cbbbbce7353c83efc10ada851 WatchSource:0}: Error finding container 50fb39d5fa4f7514890ec0bc9d55a9c8357d204cbbbbce7353c83efc10ada851: Status 404 returned error can't find the container with id 50fb39d5fa4f7514890ec0bc9d55a9c8357d204cbbbbce7353c83efc10ada851
Nov 03 01:15:00.801546 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.801521    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"]
Nov 03 01:15:00.811165 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.811137    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"]
Nov 03 01:15:00.817418 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.817393    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba959fb_3f3b_4b27_a34d_4a54fd0108a2.slice/crio-ad2da2b001a322553b8c4ed97e78d3db3a4194b42849f8f986d722080db324af WatchSource:0}: Error finding container ad2da2b001a322553b8c4ed97e78d3db3a4194b42849f8f986d722080db324af: Status 404 returned error can't find the container with id ad2da2b001a322553b8c4ed97e78d3db3a4194b42849f8f986d722080db324af
Nov 03 01:15:00.835600 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.835571    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f11cdb7_51df_46d8_9e54_0a695b213fee.slice/crio-9e52e6e60fe31d47d0c0308f1fa9da8615efe11550d09aee1e1a0aa555dc2c3b WatchSource:0}: Error finding container 9e52e6e60fe31d47d0c0308f1fa9da8615efe11550d09aee1e1a0aa555dc2c3b: Status 404 returned error can't find the container with id 9e52e6e60fe31d47d0c0308f1fa9da8615efe11550d09aee1e1a0aa555dc2c3b
Nov 03 01:15:00.843322 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.843299    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"]
Nov 03 01:15:00.847260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.847236    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"]
Nov 03 01:15:00.848568 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.848543    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936251b4_d4ac_4fba_8d1d_16429a680176.slice/crio-add4b1cab206f1ab4a5ec3fd084fb0bef3690b603d6e5e3f1eb02c5b3c635840 WatchSource:0}: Error finding container add4b1cab206f1ab4a5ec3fd084fb0bef3690b603d6e5e3f1eb02c5b3c635840: Status 404 returned error can't find the container with id add4b1cab206f1ab4a5ec3fd084fb0bef3690b603d6e5e3f1eb02c5b3c635840
Nov 03 01:15:00.909637 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.908569    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27149ef9_d3bd_4671_b3de_c9d8fc612caf.slice/crio-2c03df092c0082807b57a878c15577b7e1addddf5eef053ee5725aae8d3b00fa WatchSource:0}: Error finding container 2c03df092c0082807b57a878c15577b7e1addddf5eef053ee5725aae8d3b00fa: Status 404 returned error can't find the container with id 2c03df092c0082807b57a878c15577b7e1addddf5eef053ee5725aae8d3b00fa
Nov 03 01:15:00.911589 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.910813    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"]
Nov 03 01:15:00.919246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.919220    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"]
Nov 03 01:15:00.926204 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:00.925800    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:00.926321 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:00.926229    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5b3808_8f90_46dc_8360_5c35f82af67d.slice/crio-f84bd31336380d55b1b48fb9a34b49b7a327868aec9c41d88664ffab1a5f498f WatchSource:0}: Error finding container f84bd31336380d55b1b48fb9a34b49b7a327868aec9c41d88664ffab1a5f498f: Status 404 returned error can't find the container with id f84bd31336380d55b1b48fb9a34b49b7a327868aec9c41d88664ffab1a5f498f
Nov 03 01:15:01.337665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.337580    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:01.730532 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.730494    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" event={"ID":"6f11cdb7-51df-46d8-9e54-0a695b213fee","Type":"ContainerStarted","Data":"17a921cdaa0b732d65133916589fad54553cb63d333a998af3e8b9277003575e"}
Nov 03 01:15:01.730725 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.730563    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" event={"ID":"6f11cdb7-51df-46d8-9e54-0a695b213fee","Type":"ContainerStarted","Data":"9e52e6e60fe31d47d0c0308f1fa9da8615efe11550d09aee1e1a0aa555dc2c3b"}
Nov 03 01:15:01.732169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.732136    4550 generic.go:356] "Generic (PLEG): container finished" podID="015206dc-1fd7-4f34-a0b0-89e0255b1720" containerID="7d7e74d161507aead60b7b5d359e139abd06e38f51448cd28f409d9bd46a5c5f" exitCode=0
Nov 03 01:15:01.732284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.732219    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5" event={"ID":"015206dc-1fd7-4f34-a0b0-89e0255b1720","Type":"ContainerDied","Data":"7d7e74d161507aead60b7b5d359e139abd06e38f51448cd28f409d9bd46a5c5f"}
Nov 03 01:15:01.733784 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.733759    4550 generic.go:356] "Generic (PLEG): container finished" podID="8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" containerID="dccd77d2983fac6b423a927381ce635d398668aa60f9651793563255e16b0de2" exitCode=0
Nov 03 01:15:01.733909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.733795    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl" event={"ID":"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84","Type":"ContainerDied","Data":"dccd77d2983fac6b423a927381ce635d398668aa60f9651793563255e16b0de2"}
Nov 03 01:15:01.735366 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.735339    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerStarted","Data":"6e95f0f46c254fbe3c567fe02e444e02e7ddf540ff662cff10971f68ae3243dd"}
Nov 03 01:15:01.735470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.735374    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerStarted","Data":"f84bd31336380d55b1b48fb9a34b49b7a327868aec9c41d88664ffab1a5f498f"}
Nov 03 01:15:01.736838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.736811    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerStarted","Data":"ad39aea253678dd5601e29ff168b3eb4c749882c892cf392727ea274d34232d3"}
Nov 03 01:15:01.736968 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.736854    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerStarted","Data":"ad2da2b001a322553b8c4ed97e78d3db3a4194b42849f8f986d722080db324af"}
Nov 03 01:15:01.738413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.738392    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerStarted","Data":"bd6b056a43a02f3adecd31160afc95401818720bfdaf02bca20d0b000df670b6"}
Nov 03 01:15:01.738595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.738441    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerStarted","Data":"2c03df092c0082807b57a878c15577b7e1addddf5eef053ee5725aae8d3b00fa"}
Nov 03 01:15:01.739836 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.739815    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerStarted","Data":"856e374db7aa08614ec6e9359d4485fe66ee8d7d3493ada37f1cd6942a177393"}
Nov 03 01:15:01.739925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.739855    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerStarted","Data":"add4b1cab206f1ab4a5ec3fd084fb0bef3690b603d6e5e3f1eb02c5b3c635840"}
Nov 03 01:15:01.741214 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.741191    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerStarted","Data":"fcccf42df23cab0e47f83fc9283ec8d753cc3d9db2afaa2143eb15bb20f5d6bc"}
Nov 03 01:15:01.741302 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.741232    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerStarted","Data":"50fb39d5fa4f7514890ec0bc9d55a9c8357d204cbbbbce7353c83efc10ada851"}
Nov 03 01:15:01.775839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.775787    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" podStartSLOduration=1.7757737759999999 podStartE2EDuration="1.775773776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.773043907 +0000 UTC m=+4184.094580707" watchObservedRunningTime="2026-11-03 01:15:01.775773776 +0000 UTC m=+4184.097310574"
Nov 03 01:15:01.816243 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.816201    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" podStartSLOduration=1.816186946 podStartE2EDuration="1.816186946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.813039702 +0000 UTC m=+4184.134576502" watchObservedRunningTime="2026-11-03 01:15:01.816186946 +0000 UTC m=+4184.137723744"
Nov 03 01:15:01.894022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.893976    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" podStartSLOduration=1.89396167 podStartE2EDuration="1.89396167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.892596266 +0000 UTC m=+4184.214133084" watchObservedRunningTime="2026-11-03 01:15:01.89396167 +0000 UTC m=+4184.215498468"
Nov 03 01:15:01.932009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.931967    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" podStartSLOduration=1.931954613 podStartE2EDuration="1.931954613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.931480588 +0000 UTC m=+4184.253017402" watchObservedRunningTime="2026-11-03 01:15:01.931954613 +0000 UTC m=+4184.253491456"
Nov 03 01:15:01.972375 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:01.972318    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" podStartSLOduration=1.9723046119999998 podStartE2EDuration="1.972304612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:01.971100601 +0000 UTC m=+4184.292637407" watchObservedRunningTime="2026-11-03 01:15:01.972304612 +0000 UTC m=+4184.293841411"
Nov 03 01:15:02.049344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.049302    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" podStartSLOduration=2.049287876 podStartE2EDuration="2.049287876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.048313118 +0000 UTC m=+4184.369849919" watchObservedRunningTime="2026-11-03 01:15:02.049287876 +0000 UTC m=+4184.370824689"
Nov 03 01:15:02.758991 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.758954    4550 generic.go:356] "Generic (PLEG): container finished" podID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerID="17a921cdaa0b732d65133916589fad54553cb63d333a998af3e8b9277003575e" exitCode=60
Nov 03 01:15:02.759168 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.759044    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" event={"ID":"6f11cdb7-51df-46d8-9e54-0a695b213fee","Type":"ContainerDied","Data":"17a921cdaa0b732d65133916589fad54553cb63d333a998af3e8b9277003575e"}
Nov 03 01:15:02.759300 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.759281    4550 scope.go:122] "RemoveContainer" containerID="17a921cdaa0b732d65133916589fad54553cb63d333a998af3e8b9277003575e"
Nov 03 01:15:02.760979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.760965    4550 generic.go:356] "Generic (PLEG): container finished" podID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerID="6e95f0f46c254fbe3c567fe02e444e02e7ddf540ff662cff10971f68ae3243dd" exitCode=60
Nov 03 01:15:02.761062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.761039    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerDied","Data":"6e95f0f46c254fbe3c567fe02e444e02e7ddf540ff662cff10971f68ae3243dd"}
Nov 03 01:15:02.761539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.761519    4550 scope.go:122] "RemoveContainer" containerID="6e95f0f46c254fbe3c567fe02e444e02e7ddf540ff662cff10971f68ae3243dd"
Nov 03 01:15:02.767162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.767132    4550 generic.go:356] "Generic (PLEG): container finished" podID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerID="ad39aea253678dd5601e29ff168b3eb4c749882c892cf392727ea274d34232d3" exitCode=60
Nov 03 01:15:02.767259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.767204    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerDied","Data":"ad39aea253678dd5601e29ff168b3eb4c749882c892cf392727ea274d34232d3"}
Nov 03 01:15:02.767423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.767407    4550 scope.go:122] "RemoveContainer" containerID="ad39aea253678dd5601e29ff168b3eb4c749882c892cf392727ea274d34232d3"
Nov 03 01:15:02.768692 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.768672    4550 generic.go:356] "Generic (PLEG): container finished" podID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerID="bd6b056a43a02f3adecd31160afc95401818720bfdaf02bca20d0b000df670b6" exitCode=60
Nov 03 01:15:02.768797 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.768734    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerDied","Data":"bd6b056a43a02f3adecd31160afc95401818720bfdaf02bca20d0b000df670b6"}
Nov 03 01:15:02.769197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.769181    4550 scope.go:122] "RemoveContainer" containerID="bd6b056a43a02f3adecd31160afc95401818720bfdaf02bca20d0b000df670b6"
Nov 03 01:15:02.770943 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.770872    4550 generic.go:356] "Generic (PLEG): container finished" podID="936251b4-d4ac-4fba-8d1d-16429a680176" containerID="856e374db7aa08614ec6e9359d4485fe66ee8d7d3493ada37f1cd6942a177393" exitCode=60
Nov 03 01:15:02.770943 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.770938    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerDied","Data":"856e374db7aa08614ec6e9359d4485fe66ee8d7d3493ada37f1cd6942a177393"}
Nov 03 01:15:02.771238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.771215    4550 scope.go:122] "RemoveContainer" containerID="856e374db7aa08614ec6e9359d4485fe66ee8d7d3493ada37f1cd6942a177393"
Nov 03 01:15:02.776169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.776149    4550 generic.go:356] "Generic (PLEG): container finished" podID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerID="fcccf42df23cab0e47f83fc9283ec8d753cc3d9db2afaa2143eb15bb20f5d6bc" exitCode=60
Nov 03 01:15:02.776284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.776241    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerDied","Data":"fcccf42df23cab0e47f83fc9283ec8d753cc3d9db2afaa2143eb15bb20f5d6bc"}
Nov 03 01:15:02.776388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.776372    4550 scope.go:122] "RemoveContainer" containerID="fcccf42df23cab0e47f83fc9283ec8d753cc3d9db2afaa2143eb15bb20f5d6bc"
Nov 03 01:15:02.977507 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:02.977480    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:15:03.048829 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.048745    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:03.052237 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.052211    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca\") pod \"015206dc-1fd7-4f34-a0b0-89e0255b1720\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") "
Nov 03 01:15:03.052399 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.052310    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf\" (UniqueName: \"kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf\") pod \"015206dc-1fd7-4f34-a0b0-89e0255b1720\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") "
Nov 03 01:15:03.052494 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.052449    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca\") pod \"015206dc-1fd7-4f34-a0b0-89e0255b1720\" (UID: \"015206dc-1fd7-4f34-a0b0-89e0255b1720\") "
Nov 03 01:15:03.052871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.052850    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca" pod "015206dc-1fd7-4f34-a0b0-89e0255b1720" (UID: "015206dc-1fd7-4f34-a0b0-89e0255b1720"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:03.064464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.059597    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca" pod "015206dc-1fd7-4f34-a0b0-89e0255b1720" (UID: "015206dc-1fd7-4f34-a0b0-89e0255b1720"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:03.064464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.062878    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf" pod "015206dc-1fd7-4f34-a0b0-89e0255b1720" (UID: "015206dc-1fd7-4f34-a0b0-89e0255b1720"). InnerVolumeSpecName "kube-api-access-8l6lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:03.099635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.099602    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:15:03.153816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.153783    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca\") pod \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") "
Nov 03 01:15:03.153995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.153824    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca\") pod \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") "
Nov 03 01:15:03.153995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.153863    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997\" (UniqueName: \"kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997\") pod \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\" (UID: \"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84\") "
Nov 03 01:15:03.154151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.154113    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:03.154151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.154124    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015206dc-1fd7-4f34-a0b0-89e0255b1720-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:03.154151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.154133    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l6lf\" (UniqueName: \"kubernetes.io/projected/015206dc-1fd7-4f34-a0b0-89e0255b1720-kube-api-access-8l6lf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:03.154384 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.154223    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca" pod "8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" (UID: "8778b9c3-cee6-42d9-8ccd-ed5ddc189c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:03.160210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.160182    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997" pod "8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" (UID: "8778b9c3-cee6-42d9-8ccd-ed5ddc189c84"). InnerVolumeSpecName "kube-api-access-dm997". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:03.160403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.160184    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca" pod "8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" (UID: "8778b9c3-cee6-42d9-8ccd-ed5ddc189c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:03.254741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.254714    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:03.254741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.254738    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:03.254741 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.254747    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm997\" (UniqueName: \"kubernetes.io/projected/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84-kube-api-access-dm997\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:03.786209 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.786177    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerStarted","Data":"1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a"}
Nov 03 01:15:03.788286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.788263    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerStarted","Data":"6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff"}
Nov 03 01:15:03.790294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.790271    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerStarted","Data":"47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2"}
Nov 03 01:15:03.792184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.792156    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerStarted","Data":"49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f"}
Nov 03 01:15:03.793989 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.793968    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerStarted","Data":"f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8"}
Nov 03 01:15:03.795973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.795948    4550 generic.go:356] "Generic (PLEG): container finished" podID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerID="b629969159675ac489906d63d6377107ad565b13f7fe6e13a68fb0341675aacf" exitCode=60
Nov 03 01:15:03.796066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.796020    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" event={"ID":"6f11cdb7-51df-46d8-9e54-0a695b213fee","Type":"ContainerDied","Data":"b629969159675ac489906d63d6377107ad565b13f7fe6e13a68fb0341675aacf"}
Nov 03 01:15:03.796140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.796076    4550 scope.go:122] "RemoveContainer" containerID="17a921cdaa0b732d65133916589fad54553cb63d333a998af3e8b9277003575e"
Nov 03 01:15:03.796291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.796268    4550 scope.go:122] "RemoveContainer" containerID="b629969159675ac489906d63d6377107ad565b13f7fe6e13a68fb0341675aacf"
Nov 03 01:15:03.796560 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:03.796536    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp_odh-ai-gateway-infra(6f11cdb7-51df-46d8-9e54-0a695b213fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee"
Nov 03 01:15:03.797562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.797394    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"
Nov 03 01:15:03.797562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.797519    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5" event={"ID":"015206dc-1fd7-4f34-a0b0-89e0255b1720","Type":"ContainerDied","Data":"52f5375e771b3cccb1abc3f6afed9fbe959a29a3baabdfc7420fcf9ab8005077"}
Nov 03 01:15:03.799505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.799482    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f5375e771b3cccb1abc3f6afed9fbe959a29a3baabdfc7420fcf9ab8005077"
Nov 03 01:15:03.802122 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.802105    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"
Nov 03 01:15:03.812194 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.812166    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl" event={"ID":"8778b9c3-cee6-42d9-8ccd-ed5ddc189c84","Type":"ContainerDied","Data":"6820ed87a6de772c7aeeadeb332279511125de57464954406dc814c24fc48be3"}
Nov 03 01:15:03.812312 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:03.812216    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6820ed87a6de772c7aeeadeb332279511125de57464954406dc814c24fc48be3"
Nov 03 01:15:04.775848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.775816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:04.810061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.810034    4550 generic.go:356] "Generic (PLEG): container finished" podID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerID="1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a" exitCode=60
Nov 03 01:15:04.810212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.810110    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerDied","Data":"1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a"}
Nov 03 01:15:04.810212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.810150    4550 scope.go:122] "RemoveContainer" containerID="6e95f0f46c254fbe3c567fe02e444e02e7ddf540ff662cff10971f68ae3243dd"
Nov 03 01:15:04.810325 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.810307    4550 scope.go:122] "RemoveContainer" containerID="1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a"
Nov 03 01:15:04.810640 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:04.810611    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx_odh-ai-gateway-infra(ab5b3808-8f90-46dc-8360-5c35f82af67d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d"
Nov 03 01:15:04.812062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.812039    4550 generic.go:356] "Generic (PLEG): container finished" podID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerID="6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff" exitCode=60
Nov 03 01:15:04.812170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.812119    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerDied","Data":"6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff"}
Nov 03 01:15:04.812328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.812307    4550 scope.go:122] "RemoveContainer" containerID="6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff"
Nov 03 01:15:04.812608 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:04.812584    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr_odh-ai-gateway-infra(dba959fb-3f3b-4b27-a34d-4a54fd0108a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2"
Nov 03 01:15:04.825079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.825048    4550 generic.go:356] "Generic (PLEG): container finished" podID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerID="47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2" exitCode=60
Nov 03 01:15:04.825203 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.825174    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerDied","Data":"47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2"}
Nov 03 01:15:04.825537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.825514    4550 scope.go:122] "RemoveContainer" containerID="47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2"
Nov 03 01:15:04.825921 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:04.825897    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-9lk8s_odh-ai-gateway-infra(27149ef9-d3bd-4671-b3de-c9d8fc612caf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf"
Nov 03 01:15:04.828239 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.828214    4550 generic.go:356] "Generic (PLEG): container finished" podID="936251b4-d4ac-4fba-8d1d-16429a680176" containerID="49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f" exitCode=60
Nov 03 01:15:04.828353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.828284    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerDied","Data":"49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f"}
Nov 03 01:15:04.828608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.828590    4550 scope.go:122] "RemoveContainer" containerID="49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f"
Nov 03 01:15:04.828830 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:04.828813    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl_odh-ai-gateway-infra(936251b4-d4ac-4fba-8d1d-16429a680176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" podUID="936251b4-d4ac-4fba-8d1d-16429a680176"
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.830766    4550 generic.go:356] "Generic (PLEG): container finished" podID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerID="f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8" exitCode=60
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.830859    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerDied","Data":"f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8"}
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.831165    4550 scope.go:122] "RemoveContainer" containerID="f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8"
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:04.831393    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4_odh-ai-gateway-infra(38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f"
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.833529    4550 scope.go:122] "RemoveContainer" containerID="ad39aea253678dd5601e29ff168b3eb4c749882c892cf392727ea274d34232d3"
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.835995    4550 scope.go:122] "RemoveContainer" containerID="b629969159675ac489906d63d6377107ad565b13f7fe6e13a68fb0341675aacf"
Nov 03 01:15:04.840219 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:04.836277    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp_odh-ai-gateway-infra(6f11cdb7-51df-46d8-9e54-0a695b213fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee"
Nov 03 01:15:04.867108 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.867087    4550 scope.go:122] "RemoveContainer" containerID="bd6b056a43a02f3adecd31160afc95401818720bfdaf02bca20d0b000df670b6"
Nov 03 01:15:04.880181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.880161    4550 scope.go:122] "RemoveContainer" containerID="856e374db7aa08614ec6e9359d4485fe66ee8d7d3493ada37f1cd6942a177393"
Nov 03 01:15:04.897595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:04.897575    4550 scope.go:122] "RemoveContainer" containerID="fcccf42df23cab0e47f83fc9283ec8d753cc3d9db2afaa2143eb15bb20f5d6bc"
Nov 03 01:15:05.843873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:05.843846    4550 scope.go:122] "RemoveContainer" containerID="1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a"
Nov 03 01:15:05.844331 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:05.844105    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx_odh-ai-gateway-infra(ab5b3808-8f90-46dc-8360-5c35f82af67d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d"
Nov 03 01:15:05.845817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:05.845516    4550 scope.go:122] "RemoveContainer" containerID="6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff"
Nov 03 01:15:05.845817 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:05.845763    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr_odh-ai-gateway-infra(dba959fb-3f3b-4b27-a34d-4a54fd0108a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2"
Nov 03 01:15:05.847344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:05.847327    4550 scope.go:122] "RemoveContainer" containerID="47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2"
Nov 03 01:15:05.847544 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:05.847525    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-9lk8s_odh-ai-gateway-infra(27149ef9-d3bd-4671-b3de-c9d8fc612caf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf"
Nov 03 01:15:05.848856 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:05.848840    4550 scope.go:122] "RemoveContainer" containerID="49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f"
Nov 03 01:15:05.849041 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:05.849019    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl_odh-ai-gateway-infra(936251b4-d4ac-4fba-8d1d-16429a680176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" podUID="936251b4-d4ac-4fba-8d1d-16429a680176"
Nov 03 01:15:05.850200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:05.850183    4550 scope.go:122] "RemoveContainer" containerID="f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8"
Nov 03 01:15:05.850379 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:05.850363    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4_odh-ai-gateway-infra(38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f"
Nov 03 01:15:06.461800 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.461749    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:06.561690 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.561659    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:06.618315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618283    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"]
Nov 03 01:15:06.618805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618791    4550 cpu_manager.go:414] "containerMap: removing container" podUID="015206dc-1fd7-4f34-a0b0-89e0255b1720" containerName="revoke-keys"
Nov 03 01:15:06.618849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618806    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015206dc-1fd7-4f34-a0b0-89e0255b1720" containerName="revoke-keys"
Nov 03 01:15:06.618849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618828    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" containerName="revoke-keys"
Nov 03 01:15:06.618849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618835    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" containerName="revoke-keys"
Nov 03 01:15:06.618951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618921    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="015206dc-1fd7-4f34-a0b0-89e0255b1720" containerName="revoke-keys"
Nov 03 01:15:06.618951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.618932    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" containerName="revoke-keys"
Nov 03 01:15:06.623573 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.623556    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.640236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.640123    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:15:06.657348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.652356    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"]
Nov 03 01:15:06.672456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.672410    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:06.698630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.698603    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.698755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.698725    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.698798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.698753    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqxs\" (UniqueName: \"kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.799727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.799655    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.799727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.799691    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqxs\" (UniqueName: \"kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.799965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.799764    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.800281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.800250    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.802038 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.802011    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.819121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.819096    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqxs\" (UniqueName: \"kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs\") pod \"maas-api-revoke-sub-bc51a9ae-rc95p\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:06.945339 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:06.945305    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:07.274731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:07.274701    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:07.275229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:07.275203    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"]
Nov 03 01:15:07.866932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:07.866844    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p" event={"ID":"56a14dcc-b531-485f-9a58-8c084ed312b1","Type":"ContainerStarted","Data":"de78f22148ed96800fe63708c2592d3637d3de0129c46591f54312bcb756702e"}
Nov 03 01:15:07.866932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:07.866903    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p" event={"ID":"56a14dcc-b531-485f-9a58-8c084ed312b1","Type":"ContainerStarted","Data":"87b14d9aec6b7e43667ce8be96c8ebab31f17a64ff1a5193060c5a9e58850e8c"}
Nov 03 01:15:08.110626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:08.110584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:08.875117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:08.875081    4550 generic.go:356] "Generic (PLEG): container finished" podID="56a14dcc-b531-485f-9a58-8c084ed312b1" containerID="de78f22148ed96800fe63708c2592d3637d3de0129c46591f54312bcb756702e" exitCode=0
Nov 03 01:15:08.875352 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:08.875175    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p" event={"ID":"56a14dcc-b531-485f-9a58-8c084ed312b1","Type":"ContainerDied","Data":"de78f22148ed96800fe63708c2592d3637d3de0129c46591f54312bcb756702e"}
Nov 03 01:15:09.888703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:09.888673    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:10.046614 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.046591    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:10.144873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.144803    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs\" (UniqueName: \"kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs\") pod \"56a14dcc-b531-485f-9a58-8c084ed312b1\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") "
Nov 03 01:15:10.144873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.144870    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca\") pod \"56a14dcc-b531-485f-9a58-8c084ed312b1\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") "
Nov 03 01:15:10.145045 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.144895    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca\") pod \"56a14dcc-b531-485f-9a58-8c084ed312b1\" (UID: \"56a14dcc-b531-485f-9a58-8c084ed312b1\") "
Nov 03 01:15:10.145253 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.145231    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca" pod "56a14dcc-b531-485f-9a58-8c084ed312b1" (UID: "56a14dcc-b531-485f-9a58-8c084ed312b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:10.147012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.146983    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs" pod "56a14dcc-b531-485f-9a58-8c084ed312b1" (UID: "56a14dcc-b531-485f-9a58-8c084ed312b1"). InnerVolumeSpecName "kube-api-access-qzqxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:10.147012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.146989    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca" pod "56a14dcc-b531-485f-9a58-8c084ed312b1" (UID: "56a14dcc-b531-485f-9a58-8c084ed312b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:10.246647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.246616    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzqxs\" (UniqueName: \"kubernetes.io/projected/56a14dcc-b531-485f-9a58-8c084ed312b1-kube-api-access-qzqxs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:10.246647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.246648    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:10.246814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.246663    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56a14dcc-b531-485f-9a58-8c084ed312b1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:10.514206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.514164    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"]
Nov 03 01:15:10.514832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.514809    4550 cpu_manager.go:414] "containerMap: removing container" podUID="56a14dcc-b531-485f-9a58-8c084ed312b1" containerName="revoke-keys"
Nov 03 01:15:10.514832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.514831    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a14dcc-b531-485f-9a58-8c084ed312b1" containerName="revoke-keys"
Nov 03 01:15:10.515024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.514950    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a14dcc-b531-485f-9a58-8c084ed312b1" containerName="revoke-keys"
Nov 03 01:15:10.518280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.518258    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.536244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.536218    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"]
Nov 03 01:15:10.549102 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.549070    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk9pm\" (UniqueName: \"kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.549260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.549239    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.549475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.549449    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.647391 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.647352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:10.650837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.650808    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.650972 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.650899    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk9pm\" (UniqueName: \"kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.651041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.651009    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.651631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.651603    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.654990 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.654957    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.673763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.673733    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk9pm\" (UniqueName: \"kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm\") pod \"maas-api-revoke-sub-1d1bb7b3-h4bzb\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.832133 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.832050    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:10.981468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.977732    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p" event={"ID":"56a14dcc-b531-485f-9a58-8c084ed312b1","Type":"ContainerDied","Data":"87b14d9aec6b7e43667ce8be96c8ebab31f17a64ff1a5193060c5a9e58850e8c"}
Nov 03 01:15:10.981468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.977793    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b14d9aec6b7e43667ce8be96c8ebab31f17a64ff1a5193060c5a9e58850e8c"
Nov 03 01:15:10.981468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:10.977811    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"
Nov 03 01:15:11.148444 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:15:11.148402    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0237abd9_bc38_46c7_b3f4_a8fa462d26d8.slice/crio-d9977aa6e5f2f32d0c39bf556bbe99b97d4cb67a1a681c975f39fb31fa8a6130 WatchSource:0}: Error finding container d9977aa6e5f2f32d0c39bf556bbe99b97d4cb67a1a681c975f39fb31fa8a6130: Status 404 returned error can't find the container with id d9977aa6e5f2f32d0c39bf556bbe99b97d4cb67a1a681c975f39fb31fa8a6130
Nov 03 01:15:11.150220 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:11.150103    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"]
Nov 03 01:15:11.634547 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:11.634516    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:11.818968 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:11.818879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:11.910459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:11.910399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:11.986961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:11.986928    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb" event={"ID":"0237abd9-bc38-46c7-b3f4-a8fa462d26d8","Type":"ContainerStarted","Data":"2c5db9d9aeaed15af9b47a48e6bc06526e122c239b40d39b116db7b22b548ff8"}
Nov 03 01:15:11.987517 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:11.987483    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb" event={"ID":"0237abd9-bc38-46c7-b3f4-a8fa462d26d8","Type":"ContainerStarted","Data":"d9977aa6e5f2f32d0c39bf556bbe99b97d4cb67a1a681c975f39fb31fa8a6130"}
Nov 03 01:15:12.029184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:12.029138    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb" podStartSLOduration=2.029125678 podStartE2EDuration="2.029125678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:12.027325911 +0000 UTC m=+4194.348862729" watchObservedRunningTime="2026-11-03 01:15:12.029125678 +0000 UTC m=+4194.350662477"
Nov 03 01:15:12.513453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:12.511898    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:12.930413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:12.930384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:12.995978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:12.995945    4550 generic.go:356] "Generic (PLEG): container finished" podID="0237abd9-bc38-46c7-b3f4-a8fa462d26d8" containerID="2c5db9d9aeaed15af9b47a48e6bc06526e122c239b40d39b116db7b22b548ff8" exitCode=0
Nov 03 01:15:12.996473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:12.996009    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb" event={"ID":"0237abd9-bc38-46c7-b3f4-a8fa462d26d8","Type":"ContainerDied","Data":"2c5db9d9aeaed15af9b47a48e6bc06526e122c239b40d39b116db7b22b548ff8"}
Nov 03 01:15:13.524074 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:13.524036    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:13.811415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:13.811313    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:14.145328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.145303    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:14.212668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.212633    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm\" (UniqueName: \"kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm\") pod \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") "
Nov 03 01:15:14.212815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.212746    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca\") pod \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") "
Nov 03 01:15:14.212906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.212879    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca\") pod \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\" (UID: \"0237abd9-bc38-46c7-b3f4-a8fa462d26d8\") "
Nov 03 01:15:14.213383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.213138    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca" pod "0237abd9-bc38-46c7-b3f4-a8fa462d26d8" (UID: "0237abd9-bc38-46c7-b3f4-a8fa462d26d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:14.213538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.213421    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:14.214987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.214961    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca" pod "0237abd9-bc38-46c7-b3f4-a8fa462d26d8" (UID: "0237abd9-bc38-46c7-b3f4-a8fa462d26d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:14.215260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.215237    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm" pod "0237abd9-bc38-46c7-b3f4-a8fa462d26d8" (UID: "0237abd9-bc38-46c7-b3f4-a8fa462d26d8"). InnerVolumeSpecName "kube-api-access-lk9pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:14.314400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.314365    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:14.314400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.314400    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk9pm\" (UniqueName: \"kubernetes.io/projected/0237abd9-bc38-46c7-b3f4-a8fa462d26d8-kube-api-access-lk9pm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:14.800797 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.800762    4550 scope.go:122] "RemoveContainer" containerID="1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a"
Nov 03 01:15:14.801001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:14.800866    4550 scope.go:122] "RemoveContainer" containerID="47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2"
Nov 03 01:15:15.011399 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.011369    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"
Nov 03 01:15:15.011575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.011391    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb" event={"ID":"0237abd9-bc38-46c7-b3f4-a8fa462d26d8","Type":"ContainerDied","Data":"d9977aa6e5f2f32d0c39bf556bbe99b97d4cb67a1a681c975f39fb31fa8a6130"}
Nov 03 01:15:15.011575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.011469    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9977aa6e5f2f32d0c39bf556bbe99b97d4cb67a1a681c975f39fb31fa8a6130"
Nov 03 01:15:15.029971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.029939    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main" containerID="cri-o://85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f" gracePeriod=45
Nov 03 01:15:15.056887 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.056854    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main" containerID="cri-o://8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39" gracePeriod=45
Nov 03 01:15:15.319839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.319810    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:15.351764 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.351725    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 01:15:15.414007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.413976    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 01:15:15.427781 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.427748    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm\") pod \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") "
Nov 03 01:15:15.427934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.427823    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir\") pod \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") "
Nov 03 01:15:15.427934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.427873    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs\" (UniqueName: \"kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs\") pod \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") "
Nov 03 01:15:15.428079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.427995    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc\" (UniqueName: \"kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc\") pod \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") "
Nov 03 01:15:15.428143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.428084    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache\") pod \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") "
Nov 03 01:15:15.428143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.428127    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home\") pod \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\" (UID: \"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641\") "
Nov 03 01:15:15.428839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.428790    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache" pod "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" (UID: "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.428839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.428817    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home" pod "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" (UID: "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.429168 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.429141    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir" pod "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" (UID: "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.430772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.430744    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm" pod "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" (UID: "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.432845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.432823    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs" pod "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" (UID: "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:15.433068 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.433052    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc" pod "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" (UID: "139b7f6d-22a3-4f8a-ae1f-ebda1a5de641"). InnerVolumeSpecName "kube-api-access-vmgdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:15.529969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.529939    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs\") pod \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") "
Nov 03 01:15:15.530133 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.529995    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm\") pod \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") "
Nov 03 01:15:15.530133 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530021    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home\") pod \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") "
Nov 03 01:15:15.530133 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530074    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache\") pod \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") "
Nov 03 01:15:15.530133 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530112    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n\" (UniqueName: \"kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n\") pod \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") "
Nov 03 01:15:15.530386 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530135    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir\") pod \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\" (UID: \"c9e2d0ee-8a45-4468-afbb-c49487002bb3\") "
Nov 03 01:15:15.530386 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530354    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home" pod "c9e2d0ee-8a45-4468-afbb-c49487002bb3" (UID: "c9e2d0ee-8a45-4468-afbb-c49487002bb3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.530386 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530375    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache" pod "c9e2d0ee-8a45-4468-afbb-c49487002bb3" (UID: "c9e2d0ee-8a45-4468-afbb-c49487002bb3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.530567 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530504    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir" pod "c9e2d0ee-8a45-4468-afbb-c49487002bb3" (UID: "c9e2d0ee-8a45-4468-afbb-c49487002bb3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.530637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530605    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530618    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530628    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530635    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530643    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530652    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmgdc\" (UniqueName: \"kubernetes.io/projected/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-kube-api-access-vmgdc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530665    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530677    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.530864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.530686    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.532088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.532069    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs" pod "c9e2d0ee-8a45-4468-afbb-c49487002bb3" (UID: "c9e2d0ee-8a45-4468-afbb-c49487002bb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:15.532088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.532069    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n" pod "c9e2d0ee-8a45-4468-afbb-c49487002bb3" (UID: "c9e2d0ee-8a45-4468-afbb-c49487002bb3"). InnerVolumeSpecName "kube-api-access-tv45n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:15.532264 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.532098    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm" pod "c9e2d0ee-8a45-4468-afbb-c49487002bb3" (UID: "c9e2d0ee-8a45-4468-afbb-c49487002bb3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.632130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.632048    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e2d0ee-8a45-4468-afbb-c49487002bb3-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.632130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.632089    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e2d0ee-8a45-4468-afbb-c49487002bb3-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.632130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.632103    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv45n\" (UniqueName: \"kubernetes.io/projected/c9e2d0ee-8a45-4468-afbb-c49487002bb3-kube-api-access-tv45n\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.800604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:15.800578    4550 scope.go:122] "RemoveContainer" containerID="f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8"
Nov 03 01:15:16.022752 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.022721    4550 generic.go:356] "Generic (PLEG): container finished" podID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerID="85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f" exitCode=0
Nov 03 01:15:16.022918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.022774    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"
Nov 03 01:15:16.022918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.022808    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" event={"ID":"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641","Type":"ContainerDied","Data":"85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f"}
Nov 03 01:15:16.022918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.022851    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6" event={"ID":"139b7f6d-22a3-4f8a-ae1f-ebda1a5de641","Type":"ContainerDied","Data":"f787f7172241aaaf0f09bfdbdd8ef1fd9135c3a64ebdcf48712a2d0963f7fbe5"}
Nov 03 01:15:16.022918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.022874    4550 scope.go:122] "RemoveContainer" containerID="85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f"
Nov 03 01:15:16.025607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.025583    4550 generic.go:356] "Generic (PLEG): container finished" podID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerID="894dfbb48706137ec38e216b047832549cc8983fa936ebaafe6c465afc734e22" exitCode=60
Nov 03 01:15:16.025726 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.025688    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerDied","Data":"894dfbb48706137ec38e216b047832549cc8983fa936ebaafe6c465afc734e22"}
Nov 03 01:15:16.026118 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.026098    4550 scope.go:122] "RemoveContainer" containerID="894dfbb48706137ec38e216b047832549cc8983fa936ebaafe6c465afc734e22"
Nov 03 01:15:16.026411 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:16.026389    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx_odh-ai-gateway-infra(ab5b3808-8f90-46dc-8360-5c35f82af67d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d"
Nov 03 01:15:16.027730 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.027711    4550 generic.go:356] "Generic (PLEG): container finished" podID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerID="0252ce2a5a9177c4bc3d5812942609e49ceb92ad80979a4d6471bef85bfc3738" exitCode=60
Nov 03 01:15:16.027839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.027780    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerDied","Data":"0252ce2a5a9177c4bc3d5812942609e49ceb92ad80979a4d6471bef85bfc3738"}
Nov 03 01:15:16.028157 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.028143    4550 scope.go:122] "RemoveContainer" containerID="0252ce2a5a9177c4bc3d5812942609e49ceb92ad80979a4d6471bef85bfc3738"
Nov 03 01:15:16.028406 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:16.028385    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-9lk8s_odh-ai-gateway-infra(27149ef9-d3bd-4671-b3de-c9d8fc612caf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf"
Nov 03 01:15:16.029258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.029147    4550 generic.go:356] "Generic (PLEG): container finished" podID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerID="8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39" exitCode=0
Nov 03 01:15:16.029258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.029185    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"
Nov 03 01:15:16.029258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.029229    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" event={"ID":"c9e2d0ee-8a45-4468-afbb-c49487002bb3","Type":"ContainerDied","Data":"8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39"}
Nov 03 01:15:16.029258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.029257    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv" event={"ID":"c9e2d0ee-8a45-4468-afbb-c49487002bb3","Type":"ContainerDied","Data":"f9493d19af8347d57c45dde39b3d5f9ee174528c11bff2e4005657627f410459"}
Nov 03 01:15:16.033443 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.033261    4550 scope.go:122] "RemoveContainer" containerID="85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f"
Nov 03 01:15:16.033541 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:16.033524    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f\": container with ID starting with 85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f not found: ID does not exist" containerID="85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f"
Nov 03 01:15:16.033588 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.033548    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f"} err="failed to get container status \"85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f\": rpc error: code = NotFound desc = could not find container \"85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f\": container with ID starting with 85b639db3dfe90dc6e93a546127f275f6d78bd4331f46eb8d700b6d98ff01e1f not found: ID does not exist"
Nov 03 01:15:16.033588 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.033562    4550 scope.go:122] "RemoveContainer" containerID="1b8bc361946336aa0f8074d7fdd51c1afc59a19f0f605458c72448f97a214a8a"
Nov 03 01:15:16.046362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.046335    4550 scope.go:122] "RemoveContainer" containerID="47bb5d2e151c2ad1570a28ce0b48f386b7cba9cc4371ed35facecc52e15e49a2"
Nov 03 01:15:16.059966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.059945    4550 scope.go:122] "RemoveContainer" containerID="8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39"
Nov 03 01:15:16.068623 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.068605    4550 scope.go:122] "RemoveContainer" containerID="8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39"
Nov 03 01:15:16.068920 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:16.068889    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39\": container with ID starting with 8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39 not found: ID does not exist" containerID="8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39"
Nov 03 01:15:16.069027 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.068925    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39"} err="failed to get container status \"8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39\": rpc error: code = NotFound desc = could not find container \"8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39\": container with ID starting with 8f9b00cb9b8226d3eded35717a089ed1354495d90a45e1cf3d5beed789642a39 not found: ID does not exist"
Nov 03 01:15:16.080647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.080621    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:16.159602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.159514    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"]
Nov 03 01:15:16.164901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.164878    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/simulator-w2-4c61c8-kserve-76479cf6bd-glkbv"]
Nov 03 01:15:16.199595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.199567    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"]
Nov 03 01:15:16.202480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.202458    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/distinct-2-w2-4c61c8-kserve-7866f54b79-8pnc6"]
Nov 03 01:15:16.214724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:16.214697    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:17.042152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.042117    4550 generic.go:356] "Generic (PLEG): container finished" podID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerID="14247f2aecafc37543f6fb52d2eb7fa5de2a088a4bc303506202132e78e8b8b5" exitCode=60
Nov 03 01:15:17.042630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.042167    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerDied","Data":"14247f2aecafc37543f6fb52d2eb7fa5de2a088a4bc303506202132e78e8b8b5"}
Nov 03 01:15:17.042630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.042214    4550 scope.go:122] "RemoveContainer" containerID="f15c54de07d972df25219e4e5a6413ca3836496c11b3661ece82f667f3cba0d8"
Nov 03 01:15:17.042630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.042597    4550 scope.go:122] "RemoveContainer" containerID="14247f2aecafc37543f6fb52d2eb7fa5de2a088a4bc303506202132e78e8b8b5"
Nov 03 01:15:17.042950 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:17.042925    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4_odh-ai-gateway-infra(38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f"
Nov 03 01:15:17.095499 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.095472    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"]
Nov 03 01:15:17.140721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.140480    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"]
Nov 03 01:15:17.200806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.199420    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:17.294919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.294853    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:17.356196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.356166    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:17.458911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.458863    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca\") pod \"ab5b3808-8f90-46dc-8360-5c35f82af67d\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") "
Nov 03 01:15:17.459090 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.458933    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca\") pod \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") "
Nov 03 01:15:17.459090 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.459002    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp\" (UniqueName: \"kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp\") pod \"ab5b3808-8f90-46dc-8360-5c35f82af67d\" (UID: \"ab5b3808-8f90-46dc-8360-5c35f82af67d\") "
Nov 03 01:15:17.459090 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.459076    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts\" (UniqueName: \"kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts\") pod \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\" (UID: \"27149ef9-d3bd-4671-b3de-c9d8fc612caf\") "
Nov 03 01:15:17.459627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.459304    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca" pod "ab5b3808-8f90-46dc-8360-5c35f82af67d" (UID: "ab5b3808-8f90-46dc-8360-5c35f82af67d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:17.459627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.459449    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca" pod "27149ef9-d3bd-4671-b3de-c9d8fc612caf" (UID: "27149ef9-d3bd-4671-b3de-c9d8fc612caf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:17.460262 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.460234    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27149ef9-d3bd-4671-b3de-c9d8fc612caf-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.460387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.460267    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab5b3808-8f90-46dc-8360-5c35f82af67d-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.461730 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.461708    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts" pod "27149ef9-d3bd-4671-b3de-c9d8fc612caf" (UID: "27149ef9-d3bd-4671-b3de-c9d8fc612caf"). InnerVolumeSpecName "kube-api-access-9z4ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:17.462240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.462217    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp" pod "ab5b3808-8f90-46dc-8360-5c35f82af67d" (UID: "ab5b3808-8f90-46dc-8360-5c35f82af67d"). InnerVolumeSpecName "kube-api-access-6zvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:17.561225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.561158    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvsp\" (UniqueName: \"kubernetes.io/projected/ab5b3808-8f90-46dc-8360-5c35f82af67d-kube-api-access-6zvsp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.561225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.561190    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4ts\" (UniqueName: \"kubernetes.io/projected/27149ef9-d3bd-4671-b3de-c9d8fc612caf-kube-api-access-9z4ts\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.804753 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.804725    4550 scope.go:122] "RemoveContainer" containerID="6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff"
Nov 03 01:15:17.804919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.804833    4550 scope.go:122] "RemoveContainer" containerID="49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f"
Nov 03 01:15:17.809260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.809239    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" path="/var/lib/kubelet/pods/139b7f6d-22a3-4f8a-ae1f-ebda1a5de641/volumes"
Nov 03 01:15:17.809692 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:17.809675    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" path="/var/lib/kubelet/pods/c9e2d0ee-8a45-4468-afbb-c49487002bb3/volumes"
Nov 03 01:15:18.051883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.051856    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"
Nov 03 01:15:18.052293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.051889    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx" event={"ID":"ab5b3808-8f90-46dc-8360-5c35f82af67d","Type":"ContainerDied","Data":"f84bd31336380d55b1b48fb9a34b49b7a327868aec9c41d88664ffab1a5f498f"}
Nov 03 01:15:18.052293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.051948    4550 scope.go:122] "RemoveContainer" containerID="894dfbb48706137ec38e216b047832549cc8983fa936ebaafe6c465afc734e22"
Nov 03 01:15:18.053623 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.053602    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"
Nov 03 01:15:18.053974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.053923    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s" event={"ID":"27149ef9-d3bd-4671-b3de-c9d8fc612caf","Type":"ContainerDied","Data":"2c03df092c0082807b57a878c15577b7e1addddf5eef053ee5725aae8d3b00fa"}
Nov 03 01:15:18.063012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.062995    4550 scope.go:122] "RemoveContainer" containerID="0252ce2a5a9177c4bc3d5812942609e49ceb92ad80979a4d6471bef85bfc3738"
Nov 03 01:15:18.097932 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.097902    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"]
Nov 03 01:15:18.113540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.113515    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"]
Nov 03 01:15:18.126688 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.126658    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-9lk8s"]
Nov 03 01:15:18.167476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.166624    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"]
Nov 03 01:15:18.176040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.176013    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894475-gprmx"]
Nov 03 01:15:18.231700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.231607    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:18.369565 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.369481    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7\" (UniqueName: \"kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7\") pod \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") "
Nov 03 01:15:18.369750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.369695    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca\") pod \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\" (UID: \"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f\") "
Nov 03 01:15:18.370078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.370057    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca" pod "38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" (UID: "38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:18.371959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.371931    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7" pod "38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" (UID: "38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f"). InnerVolumeSpecName "kube-api-access-sxng7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:18.470960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.470927    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxng7\" (UniqueName: \"kubernetes.io/projected/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-kube-api-access-sxng7\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:18.470960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.470957    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:18.800828 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:18.800799    4550 scope.go:122] "RemoveContainer" containerID="b629969159675ac489906d63d6377107ad565b13f7fe6e13a68fb0341675aacf"
Nov 03 01:15:19.068839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.068798    4550 generic.go:356] "Generic (PLEG): container finished" podID="936251b4-d4ac-4fba-8d1d-16429a680176" containerID="f57c90495390b9f35e7b905d0d69d25aedfa6ca0fbd853fd1f2d8d862176ca57" exitCode=60
Nov 03 01:15:19.069231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.068892    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerDied","Data":"f57c90495390b9f35e7b905d0d69d25aedfa6ca0fbd853fd1f2d8d862176ca57"}
Nov 03 01:15:19.069231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.068946    4550 scope.go:122] "RemoveContainer" containerID="49e61436dc11549c117b2a3866250ea1b6babe563197462fcf2b50e9c4a35a0f"
Nov 03 01:15:19.069365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.069353    4550 scope.go:122] "RemoveContainer" containerID="f57c90495390b9f35e7b905d0d69d25aedfa6ca0fbd853fd1f2d8d862176ca57"
Nov 03 01:15:19.069882 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:19.069688    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl_odh-ai-gateway-infra(936251b4-d4ac-4fba-8d1d-16429a680176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" podUID="936251b4-d4ac-4fba-8d1d-16429a680176"
Nov 03 01:15:19.071098 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.071075    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4" event={"ID":"38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f","Type":"ContainerDied","Data":"50fb39d5fa4f7514890ec0bc9d55a9c8357d204cbbbbce7353c83efc10ada851"}
Nov 03 01:15:19.071188 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.071166    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"
Nov 03 01:15:19.077370 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.077342    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:19.093205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.093174    4550 generic.go:356] "Generic (PLEG): container finished" podID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerID="41fc5bae61fd1219e900bf8aea872023fe52edf7f005ccbdb2cb7a113d1d3682" exitCode=60
Nov 03 01:15:19.093461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.093242    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerDied","Data":"41fc5bae61fd1219e900bf8aea872023fe52edf7f005ccbdb2cb7a113d1d3682"}
Nov 03 01:15:19.093687 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.093671    4550 scope.go:122] "RemoveContainer" containerID="41fc5bae61fd1219e900bf8aea872023fe52edf7f005ccbdb2cb7a113d1d3682"
Nov 03 01:15:19.093960 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:19.093940    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr_odh-ai-gateway-infra(dba959fb-3f3b-4b27-a34d-4a54fd0108a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2"
Nov 03 01:15:19.097284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.097266    4550 scope.go:122] "RemoveContainer" containerID="14247f2aecafc37543f6fb52d2eb7fa5de2a088a4bc303506202132e78e8b8b5"
Nov 03 01:15:19.105959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.105937    4550 scope.go:122] "RemoveContainer" containerID="6e224eaef5222a1c36daeb606d65b703020de4bb91583acb6a6a170fe4061eff"
Nov 03 01:15:19.154817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.154788    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"]
Nov 03 01:15:19.160849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.160337    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-850433-29894475-jx8f4"]
Nov 03 01:15:19.559687 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.559651    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"]
Nov 03 01:15:19.641137 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.641104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:19.740442 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.740411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:19.817530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.815639    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" path="/var/lib/kubelet/pods/27149ef9-d3bd-4671-b3de-c9d8fc612caf/volumes"
Nov 03 01:15:19.817530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.816182    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" path="/var/lib/kubelet/pods/38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f/volumes"
Nov 03 01:15:19.817530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:19.816706    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" path="/var/lib/kubelet/pods/ab5b3808-8f90-46dc-8360-5c35f82af67d/volumes"
Nov 03 01:15:20.153510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.149752    4550 generic.go:356] "Generic (PLEG): container finished" podID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerID="9dcdcec9fd9e8c6a1d978d5591f55752c90997cb494fcd540a9f05955775feb3" exitCode=60
Nov 03 01:15:20.153510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.149978    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" event={"ID":"6f11cdb7-51df-46d8-9e54-0a695b213fee","Type":"ContainerDied","Data":"9dcdcec9fd9e8c6a1d978d5591f55752c90997cb494fcd540a9f05955775feb3"}
Nov 03 01:15:20.153510 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.150041    4550 scope.go:122] "RemoveContainer" containerID="b629969159675ac489906d63d6377107ad565b13f7fe6e13a68fb0341675aacf"
Nov 03 01:15:20.159610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.159582    4550 scope.go:122] "RemoveContainer" containerID="9dcdcec9fd9e8c6a1d978d5591f55752c90997cb494fcd540a9f05955775feb3"
Nov 03 01:15:20.160203 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:20.160177    4550 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp_odh-ai-gateway-infra(6f11cdb7-51df-46d8-9e54-0a695b213fee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee"
Nov 03 01:15:20.162646 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.162568    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"]
Nov 03 01:15:20.167241 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.167202    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:20.207472 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.207425    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"]
Nov 03 01:15:20.357479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.357424    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:20.413463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.413381    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:20.515549 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.515520    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca\") pod \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") "
Nov 03 01:15:20.515719 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.515596    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27\" (UniqueName: \"kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27\") pod \"936251b4-d4ac-4fba-8d1d-16429a680176\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") "
Nov 03 01:15:20.515719 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.515656    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c\" (UniqueName: \"kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c\") pod \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\" (UID: \"dba959fb-3f3b-4b27-a34d-4a54fd0108a2\") "
Nov 03 01:15:20.515719 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.515671    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca\") pod \"936251b4-d4ac-4fba-8d1d-16429a680176\" (UID: \"936251b4-d4ac-4fba-8d1d-16429a680176\") "
Nov 03 01:15:20.515974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.515949    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca" pod "dba959fb-3f3b-4b27-a34d-4a54fd0108a2" (UID: "dba959fb-3f3b-4b27-a34d-4a54fd0108a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:20.516051 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.516031    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca" pod "936251b4-d4ac-4fba-8d1d-16429a680176" (UID: "936251b4-d4ac-4fba-8d1d-16429a680176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:20.516139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.516124    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.517747 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.517727    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c" pod "dba959fb-3f3b-4b27-a34d-4a54fd0108a2" (UID: "dba959fb-3f3b-4b27-a34d-4a54fd0108a2"). InnerVolumeSpecName "kube-api-access-tkn8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:20.517826 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.517768    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27" pod "936251b4-d4ac-4fba-8d1d-16429a680176" (UID: "936251b4-d4ac-4fba-8d1d-16429a680176"). InnerVolumeSpecName "kube-api-access-ntg27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:20.617961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.617924    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntg27\" (UniqueName: \"kubernetes.io/projected/936251b4-d4ac-4fba-8d1d-16429a680176-kube-api-access-ntg27\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.618110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.617965    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkn8c\" (UniqueName: \"kubernetes.io/projected/dba959fb-3f3b-4b27-a34d-4a54fd0108a2-kube-api-access-tkn8c\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.618110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.617980    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/936251b4-d4ac-4fba-8d1d-16429a680176-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.780191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.780110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:20.925909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:20.925878    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:21.141226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141151    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"]
Nov 03 01:15:21.141616 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141602    4550 cpu_manager.go:414] "containerMap: removing container" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141617    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141627    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141632    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141640    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141645    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141657    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141662    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main"
Nov 03 01:15:21.141670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141672    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141676    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141691    4550 cpu_manager.go:414] "containerMap: removing container" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141697    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141706    4550 cpu_manager.go:414] "containerMap: removing container" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141710    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141717    4550 cpu_manager.go:414] "containerMap: removing container" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141721    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141730    4550 cpu_manager.go:414] "containerMap: removing container" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141735    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141742    4550 cpu_manager.go:414] "containerMap: removing container" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141746    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141753    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141757    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141763    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0237abd9-bc38-46c7-b3f4-a8fa462d26d8" containerName="revoke-keys"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141768    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0237abd9-bc38-46c7-b3f4-a8fa462d26d8" containerName="revoke-keys"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141775    4550 cpu_manager.go:414] "containerMap: removing container" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141779    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141787    4550 cpu_manager.go:414] "containerMap: removing container" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141791    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141848    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141856    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141863    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141867    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141873    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e2d0ee-8a45-4468-afbb-c49487002bb3" containerName="main"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141879    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141885    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141891    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141897    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141903    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141908    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="27149ef9-d3bd-4671-b3de-c9d8fc612caf" containerName="cleanup"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141913    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="139b7f6d-22a3-4f8a-ae1f-ebda1a5de641" containerName="main"
Nov 03 01:15:21.141907 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141918    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0237abd9-bc38-46c7-b3f4-a8fa462d26d8" containerName="revoke-keys"
Nov 03 01:15:21.142950 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.141924    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:21.145323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.145305    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.163482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.163421    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:15:21.165236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.165207    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"]
Nov 03 01:15:21.192063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.192040    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"
Nov 03 01:15:21.192063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.192044    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr" event={"ID":"dba959fb-3f3b-4b27-a34d-4a54fd0108a2","Type":"ContainerDied","Data":"ad2da2b001a322553b8c4ed97e78d3db3a4194b42849f8f986d722080db324af"}
Nov 03 01:15:21.192249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.192118    4550 scope.go:122] "RemoveContainer" containerID="41fc5bae61fd1219e900bf8aea872023fe52edf7f005ccbdb2cb7a113d1d3682"
Nov 03 01:15:21.195322 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.195289    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl" event={"ID":"936251b4-d4ac-4fba-8d1d-16429a680176","Type":"ContainerDied","Data":"add4b1cab206f1ab4a5ec3fd084fb0bef3690b603d6e5e3f1eb02c5b3c635840"}
Nov 03 01:15:21.195322 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.195319    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"
Nov 03 01:15:21.205946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.205918    4550 scope.go:122] "RemoveContainer" containerID="f57c90495390b9f35e7b905d0d69d25aedfa6ca0fbd853fd1f2d8d862176ca57"
Nov 03 01:15:21.209105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.209081    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"]
Nov 03 01:15:21.253290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.253252    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"]
Nov 03 01:15:21.263418 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.263388    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894475-vkgtl"]
Nov 03 01:15:21.327184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.327144    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.327323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.327188    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxc5\" (UniqueName: \"kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.327401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.327386    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.336263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.336240    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"]
Nov 03 01:15:21.339040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.339021    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894475-5mwhr"]
Nov 03 01:15:21.348795 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.348773    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:21.428069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.428045    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv\" (UniqueName: \"kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv\") pod \"6f11cdb7-51df-46d8-9e54-0a695b213fee\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") "
Nov 03 01:15:21.428271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.428210    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.428271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.428235    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxc5\" (UniqueName: \"kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.428414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.428341    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.429021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.428993    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.430191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.430162    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv" pod "6f11cdb7-51df-46d8-9e54-0a695b213fee" (UID: "6f11cdb7-51df-46d8-9e54-0a695b213fee"). InnerVolumeSpecName "kube-api-access-k5vvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:21.430892 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.430867    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.452067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.452042    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxc5\" (UniqueName: \"kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5\") pod \"maas-api-revoke-sub-4fc69f12-74s4l\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.458883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.458860    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:21.530009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.529631    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca\") pod \"6f11cdb7-51df-46d8-9e54-0a695b213fee\" (UID: \"6f11cdb7-51df-46d8-9e54-0a695b213fee\") "
Nov 03 01:15:21.530151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.530026    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vvv\" (UniqueName: \"kubernetes.io/projected/6f11cdb7-51df-46d8-9e54-0a695b213fee-kube-api-access-k5vvv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:21.530343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.530322    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca" pod "6f11cdb7-51df-46d8-9e54-0a695b213fee" (UID: "6f11cdb7-51df-46d8-9e54-0a695b213fee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:21.630792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.630760    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f11cdb7-51df-46d8-9e54-0a695b213fee-maas-api-service-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:21.762772 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.762730    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"]
Nov 03 01:15:21.807345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.807298    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" path="/var/lib/kubelet/pods/936251b4-d4ac-4fba-8d1d-16429a680176/volumes"
Nov 03 01:15:21.807882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:21.807863    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" path="/var/lib/kubelet/pods/dba959fb-3f3b-4b27-a34d-4a54fd0108a2/volumes"
Nov 03 01:15:22.216540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.216418    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l" event={"ID":"0ec57530-5efd-44f2-84bc-7cc817e99c2c","Type":"ContainerStarted","Data":"6ca3a602e5b20c0b17fd4ae98d39bb78b9623322218ff049756af2bc8bbe308d"}
Nov 03 01:15:22.216540 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.216519    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l" event={"ID":"0ec57530-5efd-44f2-84bc-7cc817e99c2c","Type":"ContainerStarted","Data":"295fe36b7f29a603807b21cf23432a5ae81f536987a4e8150f105eb17bcb713f"}
Nov 03 01:15:22.218530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.218507    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"
Nov 03 01:15:22.218657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.218547    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp" event={"ID":"6f11cdb7-51df-46d8-9e54-0a695b213fee","Type":"ContainerDied","Data":"9e52e6e60fe31d47d0c0308f1fa9da8615efe11550d09aee1e1a0aa555dc2c3b"}
Nov 03 01:15:22.218657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.218597    4550 scope.go:122] "RemoveContainer" containerID="9dcdcec9fd9e8c6a1d978d5591f55752c90997cb494fcd540a9f05955775feb3"
Nov 03 01:15:22.259367 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.259317    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l" podStartSLOduration=1.259305099 podStartE2EDuration="1.259305099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:22.255841483 +0000 UTC m=+4204.577378298" watchObservedRunningTime="2026-11-03 01:15:22.259305099 +0000 UTC m=+4204.580841897"
Nov 03 01:15:22.294039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.294007    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"]
Nov 03 01:15:22.296959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.296937    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ac338d-29894475-7tvqp"]
Nov 03 01:15:22.654688 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:22.654638    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:23.228146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:23.228109    4550 generic.go:356] "Generic (PLEG): container finished" podID="0ec57530-5efd-44f2-84bc-7cc817e99c2c" containerID="6ca3a602e5b20c0b17fd4ae98d39bb78b9623322218ff049756af2bc8bbe308d" exitCode=0
Nov 03 01:15:23.228651 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:23.228192    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l" event={"ID":"0ec57530-5efd-44f2-84bc-7cc817e99c2c","Type":"ContainerDied","Data":"6ca3a602e5b20c0b17fd4ae98d39bb78b9623322218ff049756af2bc8bbe308d"}
Nov 03 01:15:23.807681 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:23.807650    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" path="/var/lib/kubelet/pods/6f11cdb7-51df-46d8-9e54-0a695b213fee/volumes"
Nov 03 01:15:24.113308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.113232    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:24.326185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.326147    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:24.408613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.408580    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:24.457272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.457244    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca\") pod \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") "
Nov 03 01:15:24.457489 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.457460    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca\") pod \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") "
Nov 03 01:15:24.457579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.457504    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5\" (UniqueName: \"kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5\") pod \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\" (UID: \"0ec57530-5efd-44f2-84bc-7cc817e99c2c\") "
Nov 03 01:15:24.457886 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.457859    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca" pod "0ec57530-5efd-44f2-84bc-7cc817e99c2c" (UID: "0ec57530-5efd-44f2-84bc-7cc817e99c2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:24.458033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.457985    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:24.459453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.459405    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca" pod "0ec57530-5efd-44f2-84bc-7cc817e99c2c" (UID: "0ec57530-5efd-44f2-84bc-7cc817e99c2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:24.460054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.460029    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5" pod "0ec57530-5efd-44f2-84bc-7cc817e99c2c" (UID: "0ec57530-5efd-44f2-84bc-7cc817e99c2c"). InnerVolumeSpecName "kube-api-access-ptxc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:24.559466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.559419    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxc5\" (UniqueName: \"kubernetes.io/projected/0ec57530-5efd-44f2-84bc-7cc817e99c2c-kube-api-access-ptxc5\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:24.559466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.559466    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ec57530-5efd-44f2-84bc-7cc817e99c2c-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:24.560011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:24.559985    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerName="payload-pre-processing" containerID="cri-o://4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693" gracePeriod=25
Nov 03 01:15:24.723778 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:24.723734    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:15:25.241832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:25.241810    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"
Nov 03 01:15:25.241984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:25.241807    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l" event={"ID":"0ec57530-5efd-44f2-84bc-7cc817e99c2c","Type":"ContainerDied","Data":"295fe36b7f29a603807b21cf23432a5ae81f536987a4e8150f105eb17bcb713f"}
Nov 03 01:15:25.241984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:25.241930    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="295fe36b7f29a603807b21cf23432a5ae81f536987a4e8150f105eb17bcb713f"
Nov 03 01:15:25.938607 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:25.938575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:26.043255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:26.043220    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:27.525978 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:27.525941    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:29.015662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:29.015626    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:29.118700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:29.118668    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:30.740207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:30.740165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:32.261454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:32.260981    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:32.384843 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:32.384804    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:34.131981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:34.131948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:35.571152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:35.571112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:36.216498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:36.216465    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:37.440017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:37.439971    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:37.540059 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:37.539990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:37.801608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:37.801477    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:38.306878 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:38.306813    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:38.419706 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:38.419662    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:39.015210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:39.015174    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:39.744403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:39.744368    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:39.840569 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:39.840535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:40.027344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:40.027250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:40.470130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:40.470100    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:40.825003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:40.824912    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"]
Nov 03 01:15:40.825223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:40.825190    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" containerID="cri-o://8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80" gracePeriod=30
Nov 03 01:15:40.841909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:40.841880    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:41.620595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:41.620552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:41.932920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:41.932884    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:42.857877 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.857843    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"]
Nov 03 01:15:42.859085 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859048    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.859085 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859078    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859101    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859109    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859124    4550 cpu_manager.go:414] "containerMap: removing container" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859132    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859149    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0ec57530-5efd-44f2-84bc-7cc817e99c2c" containerName="revoke-keys"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859156    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec57530-5efd-44f2-84bc-7cc817e99c2c" containerName="revoke-keys"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859168    4550 cpu_manager.go:414] "containerMap: removing container" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859175    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859194    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859202    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859227    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.859318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859234    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859333    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec57530-5efd-44f2-84bc-7cc817e99c2c" containerName="revoke-keys"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859351    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5b3808-8f90-46dc-8360-5c35f82af67d" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859361    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba959fb-3f3b-4b27-a34d-4a54fd0108a2" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859369    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="936251b4-d4ac-4fba-8d1d-16429a680176" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859379    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859387    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859396    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:15:42.860024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.859406    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a1b0be-e2e6-43b7-bf30-77b8bb8b4a6f" containerName="cleanup"
Nov 03 01:15:42.864647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.864623    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:42.890605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.890566    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"]
Nov 03 01:15:42.925335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.925305    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:42.959470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.955901    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:15:42.959470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.955958    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:15:42.976930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:42.976889    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4c61c8-openshift-default-dockercfg-sclgr\""
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008608    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008655    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wkc\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008701    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008849    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008886    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008922    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008945    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.008968    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.009452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.009007    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.109789 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109692    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.109789 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109746    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109787    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109814    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109833    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109866    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109935    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.109965    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wkc\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.110005    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.110231    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.110582    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.110692    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.110736    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.110867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.110814    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.112354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.112320    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.112782 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.112764    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.130263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.130240    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.130470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.130445    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wkc\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.200712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.200673    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:43.438724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.438692    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:43.539456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.535589    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"]
Nov 03 01:15:43.564058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.563280    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:43.564058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.563375    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:43.564058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.563421    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:15:43.903818 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:43.903786    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:44.412362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.412325    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" event={"ID":"dc6b9040-d9f0-405a-8d47-5a6a4650c698","Type":"ContainerStarted","Data":"a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417"}
Nov 03 01:15:44.412594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.412383    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" event={"ID":"dc6b9040-d9f0-405a-8d47-5a6a4650c698","Type":"ContainerStarted","Data":"df8d8cbe25d5762df2fc2774d899821d0c4877544a332467ee0348f7d2204f2a"}
Nov 03 01:15:44.851789 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.851759    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 01:15:44.928453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.909089    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" podStartSLOduration=2.90907024 podStartE2EDuration="2.90907024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:44.495282812 +0000 UTC m=+4226.816819635" watchObservedRunningTime="2026-11-03 01:15:44.90907024 +0000 UTC m=+4227.230607039"
Nov 03 01:15:44.937343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.929827    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4\" (UniqueName: \"kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4\") pod \"658355d0-4279-4077-9d55-f38b8fb7d19b\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") "
Nov 03 01:15:44.937343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.929929    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume\") pod \"658355d0-4279-4077-9d55-f38b8fb7d19b\" (UID: \"658355d0-4279-4077-9d55-f38b8fb7d19b\") "
Nov 03 01:15:44.946893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.939671    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4" pod "658355d0-4279-4077-9d55-f38b8fb7d19b" (UID: "658355d0-4279-4077-9d55-f38b8fb7d19b"). InnerVolumeSpecName "kube-api-access-psxh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:44.946893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:44.946650    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume" pod "658355d0-4279-4077-9d55-f38b8fb7d19b" (UID: "658355d0-4279-4077-9d55-f38b8fb7d19b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:45.031264 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.031227    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxh4\" (UniqueName: \"kubernetes.io/projected/658355d0-4279-4077-9d55-f38b8fb7d19b-kube-api-access-psxh4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:45.031264 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.031261    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/658355d0-4279-4077-9d55-f38b8fb7d19b-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:45.144769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.144731    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:45.201025 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.200928    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:45.202939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.202905    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:15:45.203105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.202975    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Nov 03 01:15:45.419968 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.419931    4550 generic.go:356] "Generic (PLEG): container finished" podID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerID="4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693" exitCode=0
Nov 03 01:15:45.420176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.419975    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"
Nov 03 01:15:45.420176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.420014    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" event={"ID":"658355d0-4279-4077-9d55-f38b8fb7d19b","Type":"ContainerDied","Data":"4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693"}
Nov 03 01:15:45.420176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.420074    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw" event={"ID":"658355d0-4279-4077-9d55-f38b8fb7d19b","Type":"ContainerDied","Data":"26722ddd479c4cea0f75c3c3d3213539be5b22a7615ebe86532dcfaf5a4bf3f7"}
Nov 03 01:15:45.420176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.420103    4550 scope.go:122] "RemoveContainer" containerID="4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693"
Nov 03 01:15:45.430882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.430864    4550 scope.go:122] "RemoveContainer" containerID="4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693"
Nov 03 01:15:45.431174 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:45.431155    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693\": container with ID starting with 4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693 not found: ID does not exist" containerID="4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693"
Nov 03 01:15:45.431229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.431187    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693"} err="failed to get container status \"4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693\": rpc error: code = NotFound desc = could not find container \"4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693\": container with ID starting with 4de2da9f6a690f0096715fcc3baa4f5e64df6af88e144eacec9ab348fb33c693 not found: ID does not exist"
Nov 03 01:15:45.486966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.486846    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"]
Nov 03 01:15:45.489984 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.489958    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-850433-66886bbf87-qtlnw"]
Nov 03 01:15:45.712704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.712633    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:45.816621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.816542    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" path="/var/lib/kubelet/pods/658355d0-4279-4077-9d55-f38b8fb7d19b/volumes"
Nov 03 01:15:45.988796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:45.988757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:46.099677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.099584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:46.190508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.190478    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 01:15:46.201791 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.201761    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:15:46.201970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.201814    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Nov 03 01:15:46.245135 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245107    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245317 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245171    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245317 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245230    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245317 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245273    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245319    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245405    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245416    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:46.245512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245501    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245530    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.245740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245586    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token\") pod \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\" (UID: \"48aeed2a-e1e5-4546-93bd-94db6c9953ae\") "
Nov 03 01:15:46.246040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245578    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:46.246040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245771    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:46.246040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245689    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:46.246040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.245947    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:46.246281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.246077    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.248189 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.248163    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:46.248837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.248813    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "kube-api-access-z8nxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:46.248961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.248872    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:15:46.249048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.249031    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy" pod "48aeed2a-e1e5-4546-93bd-94db6c9953ae" (UID: "48aeed2a-e1e5-4546-93bd-94db6c9953ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:46.347009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.346973    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347005    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347016    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347025    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8nxb\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-kube-api-access-z8nxb\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347037    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347045    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347053    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.347261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.347060    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48aeed2a-e1e5-4546-93bd-94db6c9953ae-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:46.428833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.428796    4550 generic.go:356] "Generic (PLEG): container finished" podID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerID="8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80" exitCode=0
Nov 03 01:15:46.429017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.428847    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"
Nov 03 01:15:46.429017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.428878    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" event={"ID":"48aeed2a-e1e5-4546-93bd-94db6c9953ae","Type":"ContainerDied","Data":"8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80"}
Nov 03 01:15:46.429017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.428932    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp" event={"ID":"48aeed2a-e1e5-4546-93bd-94db6c9953ae","Type":"ContainerDied","Data":"622c7d83150183e89294574f8df99e6dc6b313f28feddd58257076ead93bbd0d"}
Nov 03 01:15:46.429017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.428955    4550 scope.go:122] "RemoveContainer" containerID="8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80"
Nov 03 01:15:46.439770 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.439749    4550 scope.go:122] "RemoveContainer" containerID="f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"
Nov 03 01:15:46.449310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.449291    4550 scope.go:122] "RemoveContainer" containerID="8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80"
Nov 03 01:15:46.449624 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:46.449602    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80\": container with ID starting with 8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80 not found: ID does not exist" containerID="8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80"
Nov 03 01:15:46.449733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.449637    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80"} err="failed to get container status \"8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80\": rpc error: code = NotFound desc = could not find container \"8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80\": container with ID starting with 8e49e5320bd3370bba0a3e3149412c8148bdb2e6f6f9e5fdc6b95d3705761b80 not found: ID does not exist"
Nov 03 01:15:46.449733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.449663    4550 scope.go:122] "RemoveContainer" containerID="f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"
Nov 03 01:15:46.449983 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:46.449966    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a\": container with ID starting with f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a not found: ID does not exist" containerID="f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"
Nov 03 01:15:46.450035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.449993    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a"} err="failed to get container status \"f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a\": rpc error: code = NotFound desc = could not find container \"f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a\": container with ID starting with f2d1b4a679a82e7443a5308bc589026ebed82807e60a4579d22a02184cfb954a not found: ID does not exist"
Nov 03 01:15:46.520425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.520386    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"]
Nov 03 01:15:46.525199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.525172    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-pm9dp"]
Nov 03 01:15:46.699155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.699059    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"]
Nov 03 01:15:46.782453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.781631    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:46.888385 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:46.888159    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:47.139214 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.139112    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"]
Nov 03 01:15:47.193459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.189599    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:47.201418 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.201384    4550 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:15:47.201592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.201479    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.55:15021/healthz/ready\": dial tcp 10.133.1.55:15021: connect: connection refused"
Nov 03 01:15:47.438063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.437707    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy" containerID="cri-o://a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417" gracePeriod=30
Nov 03 01:15:47.609170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.609134    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:47.808824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:47.808727    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" path="/var/lib/kubelet/pods/48aeed2a-e1e5-4546-93bd-94db6c9953ae/volumes"
Nov 03 01:15:49.334867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:49.334828    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:49.627207 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:49.627118    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:50.027973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:50.027930    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"]
Nov 03 01:15:50.062198 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:50.062155    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"]
Nov 03 01:15:50.206043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:50.206006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:50.315307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:50.315210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:50.734291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:50.734258    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:51.045715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:51.045629    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:51.250259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:51.250223    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:51.359276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:51.359193    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:51.699814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:51.699768    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn" podUID="44416698-1c57-40cc-9094-ed0e937081a7" containerName="payload-processing" containerID="cri-o://d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f" gracePeriod=25
Nov 03 01:15:51.967454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:51.967415    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 01:15:52.146010 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.145976    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z\" (UniqueName: \"kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z\") pod \"44416698-1c57-40cc-9094-ed0e937081a7\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") "
Nov 03 01:15:52.146218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.146204    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume\") pod \"44416698-1c57-40cc-9094-ed0e937081a7\" (UID: \"44416698-1c57-40cc-9094-ed0e937081a7\") "
Nov 03 01:15:52.146695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.146646    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume" pod "44416698-1c57-40cc-9094-ed0e937081a7" (UID: "44416698-1c57-40cc-9094-ed0e937081a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:52.148329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.148308    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z" pod "44416698-1c57-40cc-9094-ed0e937081a7" (UID: "44416698-1c57-40cc-9094-ed0e937081a7"). InnerVolumeSpecName "kube-api-access-5zt9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:52.247756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.247680    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44416698-1c57-40cc-9094-ed0e937081a7-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.247756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.247706    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zt9z\" (UniqueName: \"kubernetes.io/projected/44416698-1c57-40cc-9094-ed0e937081a7-kube-api-access-5zt9z\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.473100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.473056    4550 generic.go:356] "Generic (PLEG): container finished" podID="44416698-1c57-40cc-9094-ed0e937081a7" containerID="d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f" exitCode=0
Nov 03 01:15:52.473303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.473105    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"
Nov 03 01:15:52.473303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.473129    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn" event={"ID":"44416698-1c57-40cc-9094-ed0e937081a7","Type":"ContainerDied","Data":"d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f"}
Nov 03 01:15:52.473303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.473191    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn" event={"ID":"44416698-1c57-40cc-9094-ed0e937081a7","Type":"ContainerDied","Data":"fdfc2d58572a15f1bc70db45e752181d50edb1be1f7c72aa307ddfb4f4e7eddd"}
Nov 03 01:15:52.473303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.473223    4550 scope.go:122] "RemoveContainer" containerID="d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f"
Nov 03 01:15:52.484196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.484171    4550 scope.go:122] "RemoveContainer" containerID="d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f"
Nov 03 01:15:52.484528 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:52.484508    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f\": container with ID starting with d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f not found: ID does not exist" containerID="d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f"
Nov 03 01:15:52.484599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.484537    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f"} err="failed to get container status \"d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f\": rpc error: code = NotFound desc = could not find container \"d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f\": container with ID starting with d1a56db3976cb60a89334d91d82f64e12f80a6eef2adaab20f67e98fd314b47f not found: ID does not exist"
Nov 03 01:15:52.528742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.528666    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"]
Nov 03 01:15:52.540826 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.540792    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ac338d-767b8df89c-2qjtn"]
Nov 03 01:15:52.816698 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:52.816611    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:53.806445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:53.806410    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44416698-1c57-40cc-9094-ed0e937081a7" path="/var/lib/kubelet/pods/44416698-1c57-40cc-9094-ed0e937081a7/volumes"
Nov 03 01:15:54.526684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:54.526645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:54.723354 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:54.723319    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:15:55.029011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:55.028971    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerName="payload-processing" containerID="cri-o://80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018" gracePeriod=25
Nov 03 01:15:55.062652 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:55.062610    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" containerName="payload-pre-processing" containerID="cri-o://e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497" gracePeriod=25
Nov 03 01:15:55.268631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:55.268597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:56.114299 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.114212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:56.228817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.228740    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:56.305168 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305130    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305218    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305313    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305341    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305414    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305488    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305546    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305580    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.305761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305587    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.305761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.305626    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy\") pod \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\" (UID: \"dc6b9040-d9f0-405a-8d47-5a6a4650c698\") "
Nov 03 01:15:56.306051 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.306032    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.306612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.306502    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.306612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.306571    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.306612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.306585    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.306847 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.306742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:56.307967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.307940    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:15:56.308066 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.308002    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "kube-api-access-p2wkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:56.308524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.308503    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:56.309053 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.309024    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token" pod "dc6b9040-d9f0-405a-8d47-5a6a4650c698" (UID: "dc6b9040-d9f0-405a-8d47-5a6a4650c698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:56.385675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.385630    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:56.407287 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407248    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wkc\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-kube-api-access-p2wkc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407287 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407282    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407287 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407292    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407301    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407310    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407319    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407327    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.407583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.407335    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc6b9040-d9f0-405a-8d47-5a6a4650c698-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:15:56.515530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.515421    4550 generic.go:356] "Generic (PLEG): container finished" podID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerID="a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417" exitCode=0
Nov 03 01:15:56.515530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.515476    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"
Nov 03 01:15:56.515740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.515469    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" event={"ID":"dc6b9040-d9f0-405a-8d47-5a6a4650c698","Type":"ContainerDied","Data":"a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417"}
Nov 03 01:15:56.515740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.515600    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6" event={"ID":"dc6b9040-d9f0-405a-8d47-5a6a4650c698","Type":"ContainerDied","Data":"df8d8cbe25d5762df2fc2774d899821d0c4877544a332467ee0348f7d2204f2a"}
Nov 03 01:15:56.515740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.515622    4550 scope.go:122] "RemoveContainer" containerID="a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417"
Nov 03 01:15:56.526294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.526268    4550 scope.go:122] "RemoveContainer" containerID="a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417"
Nov 03 01:15:56.526672 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:15:56.526649    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417\": container with ID starting with a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417 not found: ID does not exist" containerID="a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417"
Nov 03 01:15:56.526754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.526686    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417"} err="failed to get container status \"a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417\": rpc error: code = NotFound desc = could not find container \"a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417\": container with ID starting with a03497896f376cb2303d1606bc99ffd7904ae82f416076838d10db9e3c7bb417 not found: ID does not exist"
Nov 03 01:15:56.569767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.569733    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"]
Nov 03 01:15:56.575669 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:56.575636    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-2ncv6"]
Nov 03 01:15:57.808162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:57.808127    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" path="/var/lib/kubelet/pods/dc6b9040-d9f0-405a-8d47-5a6a4650c698/volumes"
Nov 03 01:15:57.818477 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:57.818426    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:58.755298 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:58.755256    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:58.755298 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:58.755268    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:59.303678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:59.303642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:15:59.936742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:15:59.936707    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:01.456333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:01.456291    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:02.090650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:02.090608    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:02.190161 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:02.190125    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:02.831386 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:02.831351    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:03.035304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.035269    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"]
Nov 03 01:16:03.048506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.048471    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7c4a5eb-hrn7q"]
Nov 03 01:16:03.188973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.188916    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"]
Nov 03 01:16:03.189622 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.189575    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" containerID="cri-o://b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955" gracePeriod=30
Nov 03 01:16:03.214176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.214143    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:03.808622 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.808586    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d68abf-a64a-40d0-abfb-a04df36a425c" path="/var/lib/kubelet/pods/05d68abf-a64a-40d0-abfb-a04df36a425c/volumes"
Nov 03 01:16:03.839554 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:03.839507    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:04.001309 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:04.001274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:05.378746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:05.378708    4550 patch_prober.go:29] interesting pod/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:16:05.379153 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:05.378783    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:16:05.624717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:05.624684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:07.241014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:07.240980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:08.465194 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.465109    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 01:16:08.546992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.546961    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547035    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547072    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547116    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547156    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547324    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547407    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.547463 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547444    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547539    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547578    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket\") pod \"91526e27-ec44-4306-aed5-67fb95bad247\" (UID: \"91526e27-ec44-4306-aed5-67fb95bad247\") "
Nov 03 01:16:08.547752 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547673    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.548019 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.547978    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:08.548019 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.548013    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.548476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.548038    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.548476 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.548259    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.548802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.548769    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.549966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.549929    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:08.550132 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.550105    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:08.550214 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.550111    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "kube-api-access-z9pjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:08.550786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.550764    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo" pod "91526e27-ec44-4306-aed5-67fb95bad247" (UID: "91526e27-ec44-4306-aed5-67fb95bad247"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:08.605739 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.605701    4550 generic.go:356] "Generic (PLEG): container finished" podID="91526e27-ec44-4306-aed5-67fb95bad247" containerID="b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955" exitCode=0
Nov 03 01:16:08.605969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.605758    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"
Nov 03 01:16:08.605969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.605791    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" event={"ID":"91526e27-ec44-4306-aed5-67fb95bad247","Type":"ContainerDied","Data":"b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955"}
Nov 03 01:16:08.605969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.605864    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq" event={"ID":"91526e27-ec44-4306-aed5-67fb95bad247","Type":"ContainerDied","Data":"1abcfeba87eb05e55f9322fdcea1ed7f3064516fc5f966ed3842e9ad1edc4622"}
Nov 03 01:16:08.605969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.605896    4550 scope.go:122] "RemoveContainer" containerID="b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955"
Nov 03 01:16:08.616584 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.616562    4550 scope.go:122] "RemoveContainer" containerID="e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"
Nov 03 01:16:08.626863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.626839    4550 scope.go:122] "RemoveContainer" containerID="b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955"
Nov 03 01:16:08.627209 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:08.627189    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955\": container with ID starting with b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955 not found: ID does not exist" containerID="b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955"
Nov 03 01:16:08.627260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.627222    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955"} err="failed to get container status \"b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955\": rpc error: code = NotFound desc = could not find container \"b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955\": container with ID starting with b6dedf022c34571b539dd9560a38c1384d6af1c812795ca19d734f340ef92955 not found: ID does not exist"
Nov 03 01:16:08.627260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.627241    4550 scope.go:122] "RemoveContainer" containerID="e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"
Nov 03 01:16:08.627539 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:08.627508    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05\": container with ID starting with e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05 not found: ID does not exist" containerID="e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"
Nov 03 01:16:08.627617 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.627546    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05"} err="failed to get container status \"e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05\": rpc error: code = NotFound desc = could not find container \"e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05\": container with ID starting with e2c781f4c5688ffe3d022eae0efacf9a522c2b830a6f602e8fa9cf0508127c05 not found: ID does not exist"
Nov 03 01:16:08.649258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649226    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.649258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649256    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9pjf\" (UniqueName: \"kubernetes.io/projected/91526e27-ec44-4306-aed5-67fb95bad247-kube-api-access-z9pjf\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.649258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649266    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.649551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649275    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91526e27-ec44-4306-aed5-67fb95bad247-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.649551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649284    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.649551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649292    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91526e27-ec44-4306-aed5-67fb95bad247-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.649551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.649302    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91526e27-ec44-4306-aed5-67fb95bad247-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:08.661409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.661368    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"]
Nov 03 01:16:08.664280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.664253    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c738e0-openshift-default-8ff7b4997-2f2dq"]
Nov 03 01:16:08.755174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.755082    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:08.755353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.755082    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:08.874551 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:08.874511    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:09.807497 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:09.807461    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91526e27-ec44-4306-aed5-67fb95bad247" path="/var/lib/kubelet/pods/91526e27-ec44-4306-aed5-67fb95bad247/volumes"
Nov 03 01:16:10.199190 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:10.199160    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:10.303717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:10.303174    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:10.633757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:10.633721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:10.739420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:10.739378    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:10.923934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:10.923901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:11.035315 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:11.035286    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:11.630977 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:11.630939    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:11.997303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:11.997264    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"]
Nov 03 01:16:11.998110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:11.998073    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" containerID="cri-o://ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67" gracePeriod=30
Nov 03 01:16:12.017557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:12.017525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:12.070054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:12.070020    4550 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:16:12.070227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:12.070071    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:16:12.522726 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:12.522693    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:12.813804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:12.813699    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:13.038748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:13.038712    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"]
Nov 03 01:16:13.051766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:13.051735    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecd14b15-n6qlm"]
Nov 03 01:16:13.808649 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:13.808611    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6522ca-5488-47da-ba9a-3009bfed6182" path="/var/lib/kubelet/pods/5e6522ca-5488-47da-ba9a-3009bfed6182/volumes"
Nov 03 01:16:14.328856 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:14.328822    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:15.334055 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.334025    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 01:16:15.362771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.362736    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 01:16:15.419250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.418748    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume\") pod \"64f77735-43c7-4784-8e9e-87cd68607eca\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") "
Nov 03 01:16:15.419250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.418863    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8\" (UniqueName: \"kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8\") pod \"2e0f087f-fb65-4426-a283-bbb296179bf7\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") "
Nov 03 01:16:15.419250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.418947    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume\") pod \"2e0f087f-fb65-4426-a283-bbb296179bf7\" (UID: \"2e0f087f-fb65-4426-a283-bbb296179bf7\") "
Nov 03 01:16:15.419250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.418985    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9\" (UniqueName: \"kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9\") pod \"64f77735-43c7-4784-8e9e-87cd68607eca\" (UID: \"64f77735-43c7-4784-8e9e-87cd68607eca\") "
Nov 03 01:16:15.419250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.419218    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume" pod "64f77735-43c7-4784-8e9e-87cd68607eca" (UID: "64f77735-43c7-4784-8e9e-87cd68607eca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:15.419623 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.419372    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f77735-43c7-4784-8e9e-87cd68607eca-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.419749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.419725    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume" pod "2e0f087f-fb65-4426-a283-bbb296179bf7" (UID: "2e0f087f-fb65-4426-a283-bbb296179bf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:15.421250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.421222    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9" pod "64f77735-43c7-4784-8e9e-87cd68607eca" (UID: "64f77735-43c7-4784-8e9e-87cd68607eca"). InnerVolumeSpecName "kube-api-access-lddx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:15.422129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.422105    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8" pod "2e0f087f-fb65-4426-a283-bbb296179bf7" (UID: "2e0f087f-fb65-4426-a283-bbb296179bf7"). InnerVolumeSpecName "kube-api-access-5xgr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:15.520413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.520375    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xgr8\" (UniqueName: \"kubernetes.io/projected/2e0f087f-fb65-4426-a283-bbb296179bf7-kube-api-access-5xgr8\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.520413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.520408    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0f087f-fb65-4426-a283-bbb296179bf7-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.520413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.520422    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddx9\" (UniqueName: \"kubernetes.io/projected/64f77735-43c7-4784-8e9e-87cd68607eca-kube-api-access-lddx9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.649215 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.649137    4550 generic.go:356] "Generic (PLEG): container finished" podID="64f77735-43c7-4784-8e9e-87cd68607eca" containerID="e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497" exitCode=0
Nov 03 01:16:15.649215 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.649186    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"
Nov 03 01:16:15.649465 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.649214    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" event={"ID":"64f77735-43c7-4784-8e9e-87cd68607eca","Type":"ContainerDied","Data":"e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497"}
Nov 03 01:16:15.649465 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.649268    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6" event={"ID":"64f77735-43c7-4784-8e9e-87cd68607eca","Type":"ContainerDied","Data":"3331568a742a7564cf7af29e59fe04262208375911851215889d17d6c73c80b9"}
Nov 03 01:16:15.649465 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.649290    4550 scope.go:122] "RemoveContainer" containerID="e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497"
Nov 03 01:16:15.650748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.650727    4550 generic.go:356] "Generic (PLEG): container finished" podID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerID="80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018" exitCode=0
Nov 03 01:16:15.650893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.650768    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" event={"ID":"2e0f087f-fb65-4426-a283-bbb296179bf7","Type":"ContainerDied","Data":"80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018"}
Nov 03 01:16:15.650893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.650774    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"
Nov 03 01:16:15.650893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.650805    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw" event={"ID":"2e0f087f-fb65-4426-a283-bbb296179bf7","Type":"ContainerDied","Data":"a1205aca422ae46b0246110ede77d472afabe39faaaa1e3a3761f65f39f293ed"}
Nov 03 01:16:15.660151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.660135    4550 scope.go:122] "RemoveContainer" containerID="e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497"
Nov 03 01:16:15.660469 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:15.660443    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497\": container with ID starting with e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497 not found: ID does not exist" containerID="e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497"
Nov 03 01:16:15.660592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.660480    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497"} err="failed to get container status \"e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497\": rpc error: code = NotFound desc = could not find container \"e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497\": container with ID starting with e1c221b5756556b3bdb49f50fe4fab5bf289cee185cd446713a3e436d42c5497 not found: ID does not exist"
Nov 03 01:16:15.660592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.660503    4550 scope.go:122] "RemoveContainer" containerID="80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018"
Nov 03 01:16:15.670474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.670454    4550 scope.go:122] "RemoveContainer" containerID="80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018"
Nov 03 01:16:15.670761 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:15.670737    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018\": container with ID starting with 80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018 not found: ID does not exist" containerID="80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018"
Nov 03 01:16:15.670813 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.670773    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018"} err="failed to get container status \"80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018\": rpc error: code = NotFound desc = could not find container \"80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018\": container with ID starting with 80d29924178af5b74a356340bf827ba610c93533153fd3b6c4d833e375ba6018 not found: ID does not exist"
Nov 03 01:16:15.710678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.706529    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"]
Nov 03 01:16:15.716556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.716525    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c738e0-6856cdb979-52qxw"]
Nov 03 01:16:15.754888 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.754855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:15.758513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.756897    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"]
Nov 03 01:16:15.761011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.760987    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c738e0-748956ddd7-vncm6"]
Nov 03 01:16:15.808851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.808819    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" path="/var/lib/kubelet/pods/2e0f087f-fb65-4426-a283-bbb296179bf7/volumes"
Nov 03 01:16:15.809235 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:15.809221    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" path="/var/lib/kubelet/pods/64f77735-43c7-4784-8e9e-87cd68607eca/volumes"
Nov 03 01:16:16.218564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:16.218533    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:16.389966 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:16.389866    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:16.852149 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:16.852110    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:17.267647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.267618    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 01:16:17.343110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343071    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343127    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343161    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343181    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343200    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343246    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343297    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343336    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343366    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2\") pod \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\" (UID: \"96b3eb4a-d717-44b7-b0bc-18f8997e97dd\") "
Nov 03 01:16:17.343670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343414    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:17.343670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343422    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:17.343670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343547    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:17.343954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343750    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:17.343954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343877    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.343954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343895    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.343954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343908    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.343954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.343922    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.344234 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.344185    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:17.345743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.345713    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:17.345880 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.345754    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:17.346034 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.346011    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:17.346204 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.346189    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2" pod "96b3eb4a-d717-44b7-b0bc-18f8997e97dd" (UID: "96b3eb4a-d717-44b7-b0bc-18f8997e97dd"). InnerVolumeSpecName "kube-api-access-298r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:17.445176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.445141    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.445176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.445171    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.445176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.445180    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.445708 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.445189    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.445708 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.445197    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-298r2\" (UniqueName: \"kubernetes.io/projected/96b3eb4a-d717-44b7-b0bc-18f8997e97dd-kube-api-access-298r2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:17.667518 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.667482    4550 generic.go:356] "Generic (PLEG): container finished" podID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerID="ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67" exitCode=0
Nov 03 01:16:17.667702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.667548    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" event={"ID":"96b3eb4a-d717-44b7-b0bc-18f8997e97dd","Type":"ContainerDied","Data":"ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67"}
Nov 03 01:16:17.667702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.667558    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"
Nov 03 01:16:17.667702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.667615    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c" event={"ID":"96b3eb4a-d717-44b7-b0bc-18f8997e97dd","Type":"ContainerDied","Data":"025e09d05db8b9ad5cc043a2e58f05f1a5f60d9e40953cc3c66e21bca76a01eb"}
Nov 03 01:16:17.667702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.667637    4550 scope.go:122] "RemoveContainer" containerID="ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67"
Nov 03 01:16:17.677693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.677671    4550 scope.go:122] "RemoveContainer" containerID="f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"
Nov 03 01:16:17.686155 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.686135    4550 scope.go:122] "RemoveContainer" containerID="ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67"
Nov 03 01:16:17.686417 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:17.686397    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67\": container with ID starting with ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67 not found: ID does not exist" containerID="ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67"
Nov 03 01:16:17.686514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.686442    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67"} err="failed to get container status \"ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67\": rpc error: code = NotFound desc = could not find container \"ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67\": container with ID starting with ed8c352b944659c5bd562aefc57e0b6d6a0dad1e5bc6f62900388a0c3bc8ae67 not found: ID does not exist"
Nov 03 01:16:17.686514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.686467    4550 scope.go:122] "RemoveContainer" containerID="f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"
Nov 03 01:16:17.686749 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:17.686731    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410\": container with ID starting with f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410 not found: ID does not exist" containerID="f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"
Nov 03 01:16:17.686794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.686755    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410"} err="failed to get container status \"f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410\": rpc error: code = NotFound desc = could not find container \"f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410\": container with ID starting with f61f8b2accb5773a85213b4ac578974136c533dc53f1582bc13c3a2da1606410 not found: ID does not exist"
Nov 03 01:16:17.734236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.734159    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"]
Nov 03 01:16:17.747091 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.747064    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-nl82c"]
Nov 03 01:16:17.809217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:17.809188    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" path="/var/lib/kubelet/pods/96b3eb4a-d717-44b7-b0bc-18f8997e97dd/volumes"
Nov 03 01:16:18.213226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:18.213188    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:18.969801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:18.969753    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:19.082731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:19.082702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:19.474110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:19.474080    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:20.023285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:20.023252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:21.611314 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:21.611273    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:22.402661 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:22.402625    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:22.504041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:22.503997    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:23.214595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:23.214552    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:23.338839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:23.338809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:23.568131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:23.568094    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"]
Nov 03 01:16:23.568657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:23.568628    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy" containerID="cri-o://449a6396e38dfb9408d3393ce2da68f24a5e217e82b691ae0f06201a0e347c0b" gracePeriod=30
Nov 03 01:16:23.596724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:23.596688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:24.374920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:24.374887    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:24.725469 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:24.725412    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:16:24.966295 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:24.966250    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:25.131975 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:25.131872    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:26.601183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:26.601142    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:27.238298 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:27.238262    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:28.745001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.744967    4550 generic.go:356] "Generic (PLEG): container finished" podID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerID="449a6396e38dfb9408d3393ce2da68f24a5e217e82b691ae0f06201a0e347c0b" exitCode=0
Nov 03 01:16:28.745416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.745036    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" event={"ID":"05672915-3fd5-4fdd-af1b-36ad57cff170","Type":"ContainerDied","Data":"449a6396e38dfb9408d3393ce2da68f24a5e217e82b691ae0f06201a0e347c0b"}
Nov 03 01:16:28.745416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.745105    4550 scope.go:122] "RemoveContainer" containerID="1ec4bdb682700f9298dbb93ad66bab7c02add3faeb25332c8cd8cb927ff5338b"
Nov 03 01:16:28.852003 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.851938    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 01:16:28.917629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.917587    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:28.986459 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986402    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986470    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986526    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986594    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986621    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986648    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986694    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986752    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986796    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs\") pod \"05672915-3fd5-4fdd-af1b-36ad57cff170\" (UID: \"05672915-3fd5-4fdd-af1b-36ad57cff170\") "
Nov 03 01:16:28.986958 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.986920    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:28.987259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987200    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:28.987348 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987264    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:28.987497 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987474    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:28.987587 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987501    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:28.987587 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987515    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05672915-3fd5-4fdd-af1b-36ad57cff170-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:28.987863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987839    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:28.988009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.987987    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:28.989711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.989680    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "kube-api-access-7zbdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:28.989926 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.989735    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:28.990771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.990747    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:28.991133 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:28.991116    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo" pod "05672915-3fd5-4fdd-af1b-36ad57cff170" (UID: "05672915-3fd5-4fdd-af1b-36ad57cff170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:29.089069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.089024    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:29.089069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.089063    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:29.089069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.089073    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbdc\" (UniqueName: \"kubernetes.io/projected/05672915-3fd5-4fdd-af1b-36ad57cff170-kube-api-access-7zbdc\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:29.089401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.089085    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:29.089401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.089094    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:29.089401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.089101    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05672915-3fd5-4fdd-af1b-36ad57cff170-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:29.751492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.751412    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"
Nov 03 01:16:29.751492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.751441    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4" event={"ID":"05672915-3fd5-4fdd-af1b-36ad57cff170","Type":"ContainerDied","Data":"c166e460339e171352a92efa3b879dd683c0dcddf62756ca861cc5e4b1327388"}
Nov 03 01:16:29.751988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.751532    4550 scope.go:122] "RemoveContainer" containerID="449a6396e38dfb9408d3393ce2da68f24a5e217e82b691ae0f06201a0e347c0b"
Nov 03 01:16:29.819116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.819084    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"]
Nov 03 01:16:29.832483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:29.832452    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0db47e-openshift-default-ffcc9464f-vf8k4"]
Nov 03 01:16:30.575637 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:30.575597    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:30.676657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:30.676624    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:30.911926 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:30.911893    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:31.021017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:31.020978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:31.129159 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:31.129127    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:31.796574 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:31.796535    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:31.807008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:31.806978    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" path="/var/lib/kubelet/pods/05672915-3fd5-4fdd-af1b-36ad57cff170/volumes"
Nov 03 01:16:32.136505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:32.136385    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"]
Nov 03 01:16:32.137412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:32.137366    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy" containerID="cri-o://ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f" gracePeriod=30
Nov 03 01:16:32.160450 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:32.160402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:32.482345 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:32.482312    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:32.945618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:32.945586    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:34.036763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:34.036729    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"]
Nov 03 01:16:34.049980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:34.049948    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d752cb1a-bcnpm"]
Nov 03 01:16:34.314822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:34.314712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:35.806963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:35.806913    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0371001-6166-4147-b499-dfaedc21c26a" path="/var/lib/kubelet/pods/c0371001-6166-4147-b499-dfaedc21c26a/volumes"
Nov 03 01:16:36.023032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:36.022980    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:37.410044 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.409977    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 01:16:37.475717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475683    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475732    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475763    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475791    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475817    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475848    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475882    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.475935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.475931    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.477199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.476742    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:37.477199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.476992    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:37.477199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.477106    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:37.477199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.477173    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:37.477841 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.477815    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:37.479887 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.479851    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:37.479993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.479911    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "kube-api-access-8r2cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:37.480144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.480120    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:37.546487 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.546414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:37.576668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.576637    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy\") pod \"6c6f197d-8e10-4bee-a8a0-748069364c01\" (UID: \"6c6f197d-8e10-4bee-a8a0-748069364c01\") "
Nov 03 01:16:37.576969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.576949    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.576974    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.576987    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.576999    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.577012    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.577027    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r2cz\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-kube-api-access-8r2cz\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.577041    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f197d-8e10-4bee-a8a0-748069364c01-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.577069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.577054    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.578730 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.578707    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy" pod "6c6f197d-8e10-4bee-a8a0-748069364c01" (UID: "6c6f197d-8e10-4bee-a8a0-748069364c01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:37.677862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.677768    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f197d-8e10-4bee-a8a0-748069364c01-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:37.747516 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.747476    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:37.820191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.820157    4550 generic.go:356] "Generic (PLEG): container finished" podID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerID="ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f" exitCode=0
Nov 03 01:16:37.820318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.820225    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"
Nov 03 01:16:37.820318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.820238    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" event={"ID":"6c6f197d-8e10-4bee-a8a0-748069364c01","Type":"ContainerDied","Data":"ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f"}
Nov 03 01:16:37.820414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.820318    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q" event={"ID":"6c6f197d-8e10-4bee-a8a0-748069364c01","Type":"ContainerDied","Data":"41829a67d333667ec2a6ba5f87b75fbb723ae3f0b6dc08d7f8815e04d820e8e1"}
Nov 03 01:16:37.820414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.820352    4550 scope.go:122] "RemoveContainer" containerID="ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f"
Nov 03 01:16:37.830210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.830192    4550 scope.go:122] "RemoveContainer" containerID="b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"
Nov 03 01:16:37.838962 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.838942    4550 scope.go:122] "RemoveContainer" containerID="ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f"
Nov 03 01:16:37.839238 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:37.839217    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f\": container with ID starting with ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f not found: ID does not exist" containerID="ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f"
Nov 03 01:16:37.839358 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.839252    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f"} err="failed to get container status \"ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f\": rpc error: code = NotFound desc = could not find container \"ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f\": container with ID starting with ec9c6a02c427ccd4a2719e67ac2769a541e5a5e7c29b1772a0d3be736659786f not found: ID does not exist"
Nov 03 01:16:37.839358 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.839272    4550 scope.go:122] "RemoveContainer" containerID="b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"
Nov 03 01:16:37.839580 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:37.839562    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a\": container with ID starting with b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a not found: ID does not exist" containerID="b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"
Nov 03 01:16:37.839626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.839588    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a"} err="failed to get container status \"b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a\": rpc error: code = NotFound desc = could not find container \"b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a\": container with ID starting with b0bc49e4ba2c815fed53ca06add91768b5a06964f9ef625c22e0f58632941b2a not found: ID does not exist"
Nov 03 01:16:37.875930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.875897    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"]
Nov 03 01:16:37.888853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:37.888822    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9908e8-openshift-default-df58d4c65-ghg7q"]
Nov 03 01:16:38.313685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:38.313647    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:38.415987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:38.414172    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:39.346288 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:39.346252    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:39.675027 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:39.674979    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:39.793613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:39.793575    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:39.809591 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:39.809555    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" path="/var/lib/kubelet/pods/6c6f197d-8e10-4bee-a8a0-748069364c01/volumes"
Nov 03 01:16:40.331078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:40.331045    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:42.026995 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:42.026953    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:43.629933 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:43.629899    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:43.746020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:43.745982    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:43.847305 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:43.847011    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:44.564401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:44.564364    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:44.988409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:44.988375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:45.548673 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:45.548640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:45.846603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:45.846519    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:46.241720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:46.241679    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:46.348738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:46.348700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:46.472202 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:46.472156    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:46.577421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:46.577347    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:46.962620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:46.962586    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:47.413961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:47.413919    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:47.626915 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:47.626875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:47.940595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:47.940553    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:48.400594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.400563    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"]
Nov 03 01:16:48.401067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401050    4550 cpu_manager.go:414] "containerMap: removing container" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerName="payload-pre-processing"
Nov 03 01:16:48.401119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401070    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerName="payload-pre-processing"
Nov 03 01:16:48.401119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401087    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy"
Nov 03 01:16:48.401119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401096    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy"
Nov 03 01:16:48.401119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401106    4550 cpu_manager.go:414] "containerMap: removing container" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" containerName="payload-pre-processing"
Nov 03 01:16:48.401119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401110    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" containerName="payload-pre-processing"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401123    4550 cpu_manager.go:414] "containerMap: removing container" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401128    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401137    4550 cpu_manager.go:414] "containerMap: removing container" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401142    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401164    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401171    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f11cdb7-51df-46d8-9e54-0a695b213fee" containerName="cleanup"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401183    4550 cpu_manager.go:414] "containerMap: removing container" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401191    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401201    4550 cpu_manager.go:414] "containerMap: removing container" podUID="44416698-1c57-40cc-9094-ed0e937081a7" containerName="payload-processing"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401208    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44416698-1c57-40cc-9094-ed0e937081a7" containerName="payload-processing"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401224    4550 cpu_manager.go:414] "containerMap: removing container" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401231    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401242    4550 cpu_manager.go:414] "containerMap: removing container" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401247    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401257    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerName="payload-processing"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401262    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerName="payload-processing"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401269    4550 cpu_manager.go:414] "containerMap: removing container" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401274    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401281    4550 cpu_manager.go:414] "containerMap: removing container" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401285    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401303    4550 cpu_manager.go:414] "containerMap: removing container" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy"
Nov 03 01:16:48.401304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.401310    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy"
Nov 03 01:16:48.402755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402727    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy"
Nov 03 01:16:48.402868 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402778    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="658355d0-4279-4077-9d55-f38b8fb7d19b" containerName="payload-pre-processing"
Nov 03 01:16:48.402868 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402791    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy"
Nov 03 01:16:48.402983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402810    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f77735-43c7-4784-8e9e-87cd68607eca" containerName="payload-pre-processing"
Nov 03 01:16:48.402983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402926    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy"
Nov 03 01:16:48.402983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402941    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy"
Nov 03 01:16:48.402983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402950    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy"
Nov 03 01:16:48.402983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402969    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="48aeed2a-e1e5-4546-93bd-94db6c9953ae" containerName="istio-proxy"
Nov 03 01:16:48.402983 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402978    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="44416698-1c57-40cc-9094-ed0e937081a7" containerName="payload-processing"
Nov 03 01:16:48.403259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.402995    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy"
Nov 03 01:16:48.403259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.403011    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6b9040-d9f0-405a-8d47-5a6a4650c698" containerName="istio-proxy"
Nov 03 01:16:48.403259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.403019    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy"
Nov 03 01:16:48.403259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.403034    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0f087f-fb65-4426-a283-bbb296179bf7" containerName="payload-processing"
Nov 03 01:16:48.403259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.403044    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="05672915-3fd5-4fdd-af1b-36ad57cff170" containerName="istio-proxy"
Nov 03 01:16:48.412937 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.412905    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.426321 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.426284    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"]
Nov 03 01:16:48.475247 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.475208    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:16:48.475247 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.475216    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:16:48.491137 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.491102    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.491137 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.491139    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kftzl\" (UniqueName: \"kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.491404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.491236    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.591957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.591919    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.592201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.592065    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.592201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.592100    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kftzl\" (UniqueName: \"kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.592934 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.592902    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.594639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.594607    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.612396 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.612360    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kftzl\" (UniqueName: \"kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl\") pod \"maas-api-revoke-sub-ed325440-rgg2v\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:48.740286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:48.740198    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:49.034000 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:16:49.033948    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a3f5e1_372a_479a_adbc_9be2176afb02.slice/crio-8650f815e1a9501d3bf9ac52fcd9b82f049b71a20e53cb677b4d6baed84a2bef WatchSource:0}: Error finding container 8650f815e1a9501d3bf9ac52fcd9b82f049b71a20e53cb677b4d6baed84a2bef: Status 404 returned error can't find the container with id 8650f815e1a9501d3bf9ac52fcd9b82f049b71a20e53cb677b4d6baed84a2bef
Nov 03 01:16:49.036150 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:49.036121    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"]
Nov 03 01:16:49.223862 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:49.223827    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:49.917760 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:49.917712    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v" event={"ID":"66a3f5e1-372a-479a-adbc-9be2176afb02","Type":"ContainerStarted","Data":"54ddde3bd11a79ced4b2adab5bad1ed2e98ce0cdcaf44536067c20907aa674ea"}
Nov 03 01:16:49.918344 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:49.917789    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v" event={"ID":"66a3f5e1-372a-479a-adbc-9be2176afb02","Type":"ContainerStarted","Data":"8650f815e1a9501d3bf9ac52fcd9b82f049b71a20e53cb677b4d6baed84a2bef"}
Nov 03 01:16:49.958381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:49.958312    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v" podStartSLOduration=1.958295656 podStartE2EDuration="1.958295656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:49.956521137 +0000 UTC m=+4292.278057955" watchObservedRunningTime="2026-11-03 01:16:49.958295656 +0000 UTC m=+4292.279832456"
Nov 03 01:16:50.887896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:50.887855    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:50.924798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:50.924760    4550 generic.go:356] "Generic (PLEG): container finished" podID="66a3f5e1-372a-479a-adbc-9be2176afb02" containerID="54ddde3bd11a79ced4b2adab5bad1ed2e98ce0cdcaf44536067c20907aa674ea" exitCode=0
Nov 03 01:16:50.925229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:50.924842    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v" event={"ID":"66a3f5e1-372a-479a-adbc-9be2176afb02","Type":"ContainerDied","Data":"54ddde3bd11a79ced4b2adab5bad1ed2e98ce0cdcaf44536067c20907aa674ea"}
Nov 03 01:16:51.028997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:51.028957    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"]
Nov 03 01:16:51.033530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:51.033500    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5eb0e557-7d8pv"]
Nov 03 01:16:51.648038 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:51.647995    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:51.748270 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:51.748234    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:51.807349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:51.807314    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="503bac80-ef28-4562-82eb-f8ceed7ef3d6" path="/var/lib/kubelet/pods/503bac80-ef28-4562-82eb-f8ceed7ef3d6/volumes"
Nov 03 01:16:52.017941 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.017895    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:52.095052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.095019    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:52.128289 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.128258    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca\") pod \"66a3f5e1-372a-479a-adbc-9be2176afb02\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") "
Nov 03 01:16:52.128544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.128404    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl\" (UniqueName: \"kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl\") pod \"66a3f5e1-372a-479a-adbc-9be2176afb02\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") "
Nov 03 01:16:52.128544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.128518    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca\") pod \"66a3f5e1-372a-479a-adbc-9be2176afb02\" (UID: \"66a3f5e1-372a-479a-adbc-9be2176afb02\") "
Nov 03 01:16:52.128693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.128663    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca" pod "66a3f5e1-372a-479a-adbc-9be2176afb02" (UID: "66a3f5e1-372a-479a-adbc-9be2176afb02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:52.128876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.128861    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.130608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.130580    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca" pod "66a3f5e1-372a-479a-adbc-9be2176afb02" (UID: "66a3f5e1-372a-479a-adbc-9be2176afb02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:52.130731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.130647    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl" pod "66a3f5e1-372a-479a-adbc-9be2176afb02" (UID: "66a3f5e1-372a-479a-adbc-9be2176afb02"). InnerVolumeSpecName "kube-api-access-kftzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:52.230485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.230393    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kftzl\" (UniqueName: \"kubernetes.io/projected/66a3f5e1-372a-479a-adbc-9be2176afb02-kube-api-access-kftzl\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.230485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.230421    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f5e1-372a-479a-adbc-9be2176afb02-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.651103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.651067    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:52.917742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.917653    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"]
Nov 03 01:16:52.929427 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.929394    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"]
Nov 03 01:16:52.934980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.934952    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"]
Nov 03 01:16:52.941127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.941093    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v" event={"ID":"66a3f5e1-372a-479a-adbc-9be2176afb02","Type":"ContainerDied","Data":"8650f815e1a9501d3bf9ac52fcd9b82f049b71a20e53cb677b4d6baed84a2bef"}
Nov 03 01:16:52.941307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.941145    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"
Nov 03 01:16:52.941537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.941368    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8650f815e1a9501d3bf9ac52fcd9b82f049b71a20e53cb677b4d6baed84a2bef"
Nov 03 01:16:52.943790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:52.943765    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"]
Nov 03 01:16:53.029243 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:53.029209    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:53.756449 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:53.756399    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:54.099302 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:54.099213    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:54.726602 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:16:54.726562    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:16:54.734458 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:54.734398    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:56.446733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:56.446698    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:57.333728 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:57.333687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:57.441049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:57.441017    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:57.909355 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:57.909319    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:58.013486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:58.013452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:58.230369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:58.230270    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:58.671525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:58.671394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:59.044873 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:59.044642    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"]
Nov 03 01:16:59.045624 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:59.044916    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy" containerID="cri-o://a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b" gracePeriod=30
Nov 03 01:16:59.078087 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:59.078051    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:59.567906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:59.567867    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:16:59.936700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:16:59.932096    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:00.034632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.034596    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:00.179325 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179285    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"]
Nov 03 01:17:00.179771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179696    4550 cpu_manager.go:414] "containerMap: removing container" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy"
Nov 03 01:17:00.179771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179709    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6f197d-8e10-4bee-a8a0-748069364c01" containerName="istio-proxy"
Nov 03 01:17:00.179771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179725    4550 cpu_manager.go:414] "containerMap: removing container" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy"
Nov 03 01:17:00.179771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179732    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b3eb4a-d717-44b7-b0bc-18f8997e97dd" containerName="istio-proxy"
Nov 03 01:17:00.179771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179750    4550 cpu_manager.go:414] "containerMap: removing container" podUID="66a3f5e1-372a-479a-adbc-9be2176afb02" containerName="revoke-keys"
Nov 03 01:17:00.179771 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179755    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a3f5e1-372a-479a-adbc-9be2176afb02" containerName="revoke-keys"
Nov 03 01:17:00.180013 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179833    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a3f5e1-372a-479a-adbc-9be2176afb02" containerName="revoke-keys"
Nov 03 01:17:00.180013 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.179843    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="91526e27-ec44-4306-aed5-67fb95bad247" containerName="istio-proxy"
Nov 03 01:17:00.182918 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.182900    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.212847 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.212817    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:17:00.213033 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.212851    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:17:00.215506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.215485    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"]
Nov 03 01:17:00.298735 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.298696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:00.316867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.316833    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.317039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.316884    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg86p\" (UniqueName: \"kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.317039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.317020    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.417832 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.417800    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.418052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.417884    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.418052 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.417926    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg86p\" (UniqueName: \"kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.418602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.418572    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.420277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.420256    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.438170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.438141    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg86p\" (UniqueName: \"kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p\") pod \"maas-api-revoke-sub-fd658052-5cvth\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.497164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.497073    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:00.611783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.611751    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:00.798581 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:17:00.798551    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3eb199b_763b_44a6_99d4_2074beac9367.slice/crio-df389555d51167943c1de6411bc9d239e0211c8f06bd2b58f372a2cfb98778f9 WatchSource:0}: Error finding container df389555d51167943c1de6411bc9d239e0211c8f06bd2b58f372a2cfb98778f9: Status 404 returned error can't find the container with id df389555d51167943c1de6411bc9d239e0211c8f06bd2b58f372a2cfb98778f9
Nov 03 01:17:00.799689 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.799661    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"]
Nov 03 01:17:00.996601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:00.996557    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth" event={"ID":"b3eb199b-763b-44a6-99d4-2074beac9367","Type":"ContainerStarted","Data":"df389555d51167943c1de6411bc9d239e0211c8f06bd2b58f372a2cfb98778f9"}
Nov 03 01:17:02.004448 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:02.004399    4550 generic.go:356] "Generic (PLEG): container finished" podID="b3eb199b-763b-44a6-99d4-2074beac9367" containerID="2d0e9efec3c8500e41411c418b2bbb7596b0b6752e56337b5ca5da58fbcaad20" exitCode=0
Nov 03 01:17:02.004864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:02.004539    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth" event={"ID":"b3eb199b-763b-44a6-99d4-2074beac9367","Type":"ContainerDied","Data":"2d0e9efec3c8500e41411c418b2bbb7596b0b6752e56337b5ca5da58fbcaad20"}
Nov 03 01:17:02.019777 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:02.019747    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:03.143796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.143765    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:03.250123 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.250084    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca\") pod \"b3eb199b-763b-44a6-99d4-2074beac9367\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") "
Nov 03 01:17:03.250328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.250161    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca\") pod \"b3eb199b-763b-44a6-99d4-2074beac9367\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") "
Nov 03 01:17:03.250328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.250249    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p\" (UniqueName: \"kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p\") pod \"b3eb199b-763b-44a6-99d4-2074beac9367\" (UID: \"b3eb199b-763b-44a6-99d4-2074beac9367\") "
Nov 03 01:17:03.250480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.250459    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca" pod "b3eb199b-763b-44a6-99d4-2074beac9367" (UID: "b3eb199b-763b-44a6-99d4-2074beac9367"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:03.250608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.250585    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.252380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.252356    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca" pod "b3eb199b-763b-44a6-99d4-2074beac9367" (UID: "b3eb199b-763b-44a6-99d4-2074beac9367"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:03.252495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.252404    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p" pod "b3eb199b-763b-44a6-99d4-2074beac9367" (UID: "b3eb199b-763b-44a6-99d4-2074beac9367"). InnerVolumeSpecName "kube-api-access-bg86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:03.351881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.351783    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg86p\" (UniqueName: \"kubernetes.io/projected/b3eb199b-763b-44a6-99d4-2074beac9367-kube-api-access-bg86p\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.351881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.351819    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3eb199b-763b-44a6-99d4-2074beac9367-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.726023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:03.725985    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:04.015924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.015843    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"
Nov 03 01:17:04.016117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.015840    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth" event={"ID":"b3eb199b-763b-44a6-99d4-2074beac9367","Type":"ContainerDied","Data":"df389555d51167943c1de6411bc9d239e0211c8f06bd2b58f372a2cfb98778f9"}
Nov 03 01:17:04.016117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.015974    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df389555d51167943c1de6411bc9d239e0211c8f06bd2b58f372a2cfb98778f9"
Nov 03 01:17:04.313484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.313456    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:17:04.465722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.465690    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.465939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.465769    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.465939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.465802    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.465939 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.465876    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.466121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.465952    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.466121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466005    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.466121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466029    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.466121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466062    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.466121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466091    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.466383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466126    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token\") pod \"424b73b2-0601-44b7-8aa2-20734d0aa15a\" (UID: \"424b73b2-0601-44b7-8aa2-20734d0aa15a\") "
Nov 03 01:17:04.466383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466042    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.466555 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466534    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.466618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466563    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.466618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466568    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.466783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466752    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:04.466863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.466833    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.468662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.468621    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:04.469062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.469038    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:04.469195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.469169    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "kube-api-access-bll4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:04.469814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.469789    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy" pod "424b73b2-0601-44b7-8aa2-20734d0aa15a" (UID: "424b73b2-0601-44b7-8aa2-20734d0aa15a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.565059 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.564973    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"]
Nov 03 01:17:04.567113 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567083    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.567271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567114    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bll4z\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-kube-api-access-bll4z\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.567271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567131    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.567271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567143    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.567271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567154    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/424b73b2-0601-44b7-8aa2-20734d0aa15a-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.567271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567166    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/424b73b2-0601-44b7-8aa2-20734d0aa15a-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.567271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.567177    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/424b73b2-0601-44b7-8aa2-20734d0aa15a-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.644940 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:04.644891    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:05.023514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.023482    4550 generic.go:356] "Generic (PLEG): container finished" podID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerID="a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b" exitCode=0
Nov 03 01:17:05.023714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.023537    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"
Nov 03 01:17:05.023714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.023573    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" event={"ID":"424b73b2-0601-44b7-8aa2-20734d0aa15a","Type":"ContainerDied","Data":"a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b"}
Nov 03 01:17:05.023714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.023633    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87" event={"ID":"424b73b2-0601-44b7-8aa2-20734d0aa15a","Type":"ContainerDied","Data":"d4e38ac8ae30d1dce81a026624142cf9263586698e9ea395ecf6b23f241b93c1"}
Nov 03 01:17:05.023714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.023663    4550 scope.go:122] "RemoveContainer" containerID="a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b"
Nov 03 01:17:05.034360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.034338    4550 scope.go:122] "RemoveContainer" containerID="a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b"
Nov 03 01:17:05.034664 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:05.034640    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b\": container with ID starting with a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b not found: ID does not exist" containerID="a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b"
Nov 03 01:17:05.034753 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.034672    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b"} err="failed to get container status \"a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b\": rpc error: code = NotFound desc = could not find container \"a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b\": container with ID starting with a8e1d5e1aea843b927d8b7174baa16bd939cc4e499493fdc9011bec1b49c473b not found: ID does not exist"
Nov 03 01:17:05.081620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.081584    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"]
Nov 03 01:17:05.094501 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.094473    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-w5z87"]
Nov 03 01:17:05.368525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.368329    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:05.534521 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.534486    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:05.807249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:05.807218    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" path="/var/lib/kubelet/pods/424b73b2-0601-44b7-8aa2-20734d0aa15a/volumes"
Nov 03 01:17:06.811802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:06.811765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:06.911717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:06.911684    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:07.223446 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.223396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:07.499745 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.499647    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:07.713630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.713589    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:07.940234 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.940192    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main" containerID="cri-o://e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274" gracePeriod=45
Nov 03 01:17:07.953720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.953678    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main" containerID="cri-o://38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4" gracePeriod=45
Nov 03 01:17:07.958815 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.958784    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main" containerID="cri-o://0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1" gracePeriod=45
Nov 03 01:17:07.970224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:07.970181    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main" containerID="cri-o://285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128" gracePeriod=45
Nov 03 01:17:08.272935 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.272906    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 01:17:08.309548 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.309517    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 01:17:08.311599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.311573    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 01:17:08.337453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.337419    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 01:17:08.419007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.418976    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs\" (UniqueName: \"kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs\") pod \"17326228-95bb-449c-ac56-919975c8b94d\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") "
Nov 03 01:17:08.419206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419021    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm\") pod \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") "
Nov 03 01:17:08.419206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419045    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir\") pod \"17326228-95bb-449c-ac56-919975c8b94d\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") "
Nov 03 01:17:08.419206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419085    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs\" (UniqueName: \"kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs\") pod \"c70820b0-0954-433d-a823-e8997a45a058\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") "
Nov 03 01:17:08.419206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419118    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir\") pod \"c70820b0-0954-433d-a823-e8997a45a058\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") "
Nov 03 01:17:08.419206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419154    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache\") pod \"c70820b0-0954-433d-a823-e8997a45a058\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") "
Nov 03 01:17:08.419206 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419196    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn\" (UniqueName: \"kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn\") pod \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419224    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home\") pod \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419253    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home\") pod \"c70820b0-0954-433d-a823-e8997a45a058\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419310    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv\" (UniqueName: \"kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv\") pod \"17326228-95bb-449c-ac56-919975c8b94d\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419387    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm\") pod \"17326228-95bb-449c-ac56-919975c8b94d\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419424    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home\") pod \"17326228-95bb-449c-ac56-919975c8b94d\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419475    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn\" (UniqueName: \"kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn\") pod \"c70820b0-0954-433d-a823-e8997a45a058\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419513    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache\") pod \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") "
Nov 03 01:17:08.419571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419567    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm\") pod \"c70820b0-0954-433d-a823-e8997a45a058\" (UID: \"c70820b0-0954-433d-a823-e8997a45a058\") "
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419592    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs\" (UniqueName: \"kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs\") pod \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") "
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419633    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache\") pod \"17326228-95bb-449c-ac56-919975c8b94d\" (UID: \"17326228-95bb-449c-ac56-919975c8b94d\") "
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419626    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache" pod "c70820b0-0954-433d-a823-e8997a45a058" (UID: "c70820b0-0954-433d-a823-e8997a45a058"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419655    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir" pod "c70820b0-0954-433d-a823-e8997a45a058" (UID: "c70820b0-0954-433d-a823-e8997a45a058"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419692    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home" pod "458c7292-7eae-491a-a1c1-4a4bbabe4ea1" (UID: "458c7292-7eae-491a-a1c1-4a4bbabe4ea1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419663    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir\") pod \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\" (UID: \"458c7292-7eae-491a-a1c1-4a4bbabe4ea1\") "
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419840    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir" pod "458c7292-7eae-491a-a1c1-4a4bbabe4ea1" (UID: "458c7292-7eae-491a-a1c1-4a4bbabe4ea1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.420035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.419898    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir" pod "17326228-95bb-449c-ac56-919975c8b94d" (UID: "17326228-95bb-449c-ac56-919975c8b94d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420121    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache" pod "458c7292-7eae-491a-a1c1-4a4bbabe4ea1" (UID: "458c7292-7eae-491a-a1c1-4a4bbabe4ea1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420204    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420228    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420241    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420254    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420268    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.420466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.420280    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.421605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.421381    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home" pod "c70820b0-0954-433d-a823-e8997a45a058" (UID: "c70820b0-0954-433d-a823-e8997a45a058"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.422974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.422799    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home" pod "17326228-95bb-449c-ac56-919975c8b94d" (UID: "17326228-95bb-449c-ac56-919975c8b94d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.423268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.423161    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache" pod "17326228-95bb-449c-ac56-919975c8b94d" (UID: "17326228-95bb-449c-ac56-919975c8b94d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.423268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.423235    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs" pod "c70820b0-0954-433d-a823-e8997a45a058" (UID: "c70820b0-0954-433d-a823-e8997a45a058"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:08.423580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.423553    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv" pod "17326228-95bb-449c-ac56-919975c8b94d" (UID: "17326228-95bb-449c-ac56-919975c8b94d"). InnerVolumeSpecName "kube-api-access-z7dnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:08.424901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.424873    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm" pod "17326228-95bb-449c-ac56-919975c8b94d" (UID: "17326228-95bb-449c-ac56-919975c8b94d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.425024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.424997    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn" pod "c70820b0-0954-433d-a823-e8997a45a058" (UID: "c70820b0-0954-433d-a823-e8997a45a058"). InnerVolumeSpecName "kube-api-access-tjkvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:08.425024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.425008    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs" pod "17326228-95bb-449c-ac56-919975c8b94d" (UID: "17326228-95bb-449c-ac56-919975c8b94d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:08.425160 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.425119    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm" pod "458c7292-7eae-491a-a1c1-4a4bbabe4ea1" (UID: "458c7292-7eae-491a-a1c1-4a4bbabe4ea1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.425422 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.425399    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn" pod "458c7292-7eae-491a-a1c1-4a4bbabe4ea1" (UID: "458c7292-7eae-491a-a1c1-4a4bbabe4ea1"). InnerVolumeSpecName "kube-api-access-zzvkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:08.426014 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.425992    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm" pod "c70820b0-0954-433d-a823-e8997a45a058" (UID: "c70820b0-0954-433d-a823-e8997a45a058"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.426227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.426210    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs" pod "458c7292-7eae-491a-a1c1-4a4bbabe4ea1" (UID: "458c7292-7eae-491a-a1c1-4a4bbabe4ea1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:08.521020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.520938    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home\") pod \"c8a89531-cdc1-4e23-bf99-3235a955c60f\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") "
Nov 03 01:17:08.521191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521025    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir\") pod \"c8a89531-cdc1-4e23-bf99-3235a955c60f\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") "
Nov 03 01:17:08.521191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521061    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache\") pod \"c8a89531-cdc1-4e23-bf99-3235a955c60f\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") "
Nov 03 01:17:08.521191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521129    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs\") pod \"c8a89531-cdc1-4e23-bf99-3235a955c60f\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") "
Nov 03 01:17:08.521191 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521162    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw\" (UniqueName: \"kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw\") pod \"c8a89531-cdc1-4e23-bf99-3235a955c60f\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") "
Nov 03 01:17:08.521425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521251    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm\") pod \"c8a89531-cdc1-4e23-bf99-3235a955c60f\" (UID: \"c8a89531-cdc1-4e23-bf99-3235a955c60f\") "
Nov 03 01:17:08.521425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521263    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home" pod "c8a89531-cdc1-4e23-bf99-3235a955c60f" (UID: "c8a89531-cdc1-4e23-bf99-3235a955c60f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.521425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521333    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir" pod "c8a89531-cdc1-4e23-bf99-3235a955c60f" (UID: "c8a89531-cdc1-4e23-bf99-3235a955c60f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.521425 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521410    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache" pod "c8a89531-cdc1-4e23-bf99-3235a955c60f" (UID: "c8a89531-cdc1-4e23-bf99-3235a955c60f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.521667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521581    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521606    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521626    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521642    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17326228-95bb-449c-ac56-919975c8b94d-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521655    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521667    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521679    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70820b0-0954-433d-a823-e8997a45a058-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521689    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521701    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521715    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzvkn\" (UniqueName: \"kubernetes.io/projected/458c7292-7eae-491a-a1c1-4a4bbabe4ea1-kube-api-access-zzvkn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521728    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70820b0-0954-433d-a823-e8997a45a058-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521741    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7dnv\" (UniqueName: \"kubernetes.io/projected/17326228-95bb-449c-ac56-919975c8b94d-kube-api-access-z7dnv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521753    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521765    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17326228-95bb-449c-ac56-919975c8b94d-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.521883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.521777    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjkvn\" (UniqueName: \"kubernetes.io/projected/c70820b0-0954-433d-a823-e8997a45a058-kube-api-access-tjkvn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.523332 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.523311    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm" pod "c8a89531-cdc1-4e23-bf99-3235a955c60f" (UID: "c8a89531-cdc1-4e23-bf99-3235a955c60f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:08.523421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.523335    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw" pod "c8a89531-cdc1-4e23-bf99-3235a955c60f" (UID: "c8a89531-cdc1-4e23-bf99-3235a955c60f"). InnerVolumeSpecName "kube-api-access-hhmcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:08.523421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.523360    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs" pod "c8a89531-cdc1-4e23-bf99-3235a955c60f" (UID: "c8a89531-cdc1-4e23-bf99-3235a955c60f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:08.622537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.622498    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8a89531-cdc1-4e23-bf99-3235a955c60f-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.622537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.622530    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8a89531-cdc1-4e23-bf99-3235a955c60f-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.622762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.622545    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhmcw\" (UniqueName: \"kubernetes.io/projected/c8a89531-cdc1-4e23-bf99-3235a955c60f-kube-api-access-hhmcw\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:08.829735 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:08.829640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:09.050662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.050628    4550 generic.go:356] "Generic (PLEG): container finished" podID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerID="285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128" exitCode=0
Nov 03 01:17:09.051140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.050681    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"
Nov 03 01:17:09.051140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.050726    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" event={"ID":"c8a89531-cdc1-4e23-bf99-3235a955c60f","Type":"ContainerDied","Data":"285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128"}
Nov 03 01:17:09.051140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.050773    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz" event={"ID":"c8a89531-cdc1-4e23-bf99-3235a955c60f","Type":"ContainerDied","Data":"320e4769083fba633a31f982b547559f90b7c9b551668a7e205c348b621859de"}
Nov 03 01:17:09.051140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.050796    4550 scope.go:122] "RemoveContainer" containerID="285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128"
Nov 03 01:17:09.052240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.052216    4550 generic.go:356] "Generic (PLEG): container finished" podID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerID="e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274" exitCode=0
Nov 03 01:17:09.052335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.052263    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"
Nov 03 01:17:09.052335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.052297    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" event={"ID":"458c7292-7eae-491a-a1c1-4a4bbabe4ea1","Type":"ContainerDied","Data":"e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274"}
Nov 03 01:17:09.052335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.052332    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth" event={"ID":"458c7292-7eae-491a-a1c1-4a4bbabe4ea1","Type":"ContainerDied","Data":"62afc6724f8e62bdba16a8a9e81916c13b4484f85cc2ddb526c2f3790f82e40a"}
Nov 03 01:17:09.053875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.053852    4550 generic.go:356] "Generic (PLEG): container finished" podID="17326228-95bb-449c-ac56-919975c8b94d" containerID="0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1" exitCode=0
Nov 03 01:17:09.054023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.053887    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"
Nov 03 01:17:09.054023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.053929    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" event={"ID":"17326228-95bb-449c-ac56-919975c8b94d","Type":"ContainerDied","Data":"0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1"}
Nov 03 01:17:09.054023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.053972    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj" event={"ID":"17326228-95bb-449c-ac56-919975c8b94d","Type":"ContainerDied","Data":"9274d459ffec47f150e0d3d72a5129b6e8eb3d654f83b42b9bf3085be0751945"}
Nov 03 01:17:09.055468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.055418    4550 generic.go:356] "Generic (PLEG): container finished" podID="c70820b0-0954-433d-a823-e8997a45a058" containerID="38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4" exitCode=0
Nov 03 01:17:09.055544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.055480    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"
Nov 03 01:17:09.055643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.055621    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" event={"ID":"c70820b0-0954-433d-a823-e8997a45a058","Type":"ContainerDied","Data":"38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4"}
Nov 03 01:17:09.055734 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.055664    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv" event={"ID":"c70820b0-0954-433d-a823-e8997a45a058","Type":"ContainerDied","Data":"28a6fe5cd8e7fa05f3ba41c40dd9e57d269df4529239b148b8675a62f9aca43b"}
Nov 03 01:17:09.061921 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.061894    4550 scope.go:122] "RemoveContainer" containerID="285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128"
Nov 03 01:17:09.062253 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:09.062227    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128\": container with ID starting with 285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128 not found: ID does not exist" containerID="285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128"
Nov 03 01:17:09.062343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.062265    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128"} err="failed to get container status \"285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128\": rpc error: code = NotFound desc = could not find container \"285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128\": container with ID starting with 285814b7dcde4d781b5bd44aed358921ee4f1b99b1925ec6f6434efc68a93128 not found: ID does not exist"
Nov 03 01:17:09.062343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.062286    4550 scope.go:122] "RemoveContainer" containerID="e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274"
Nov 03 01:17:09.073874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.073825    4550 scope.go:122] "RemoveContainer" containerID="e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274"
Nov 03 01:17:09.074241 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:09.074221    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274\": container with ID starting with e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274 not found: ID does not exist" containerID="e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274"
Nov 03 01:17:09.074338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.074251    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274"} err="failed to get container status \"e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274\": rpc error: code = NotFound desc = could not find container \"e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274\": container with ID starting with e9f9c80daa1d0cef8cbe33ecdb7e2392b7bec731e98767e1baa5d5d46a070274 not found: ID does not exist"
Nov 03 01:17:09.074338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.074273    4550 scope.go:122] "RemoveContainer" containerID="0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1"
Nov 03 01:17:09.087062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.087041    4550 scope.go:122] "RemoveContainer" containerID="0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1"
Nov 03 01:17:09.087395 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:09.087370    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1\": container with ID starting with 0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1 not found: ID does not exist" containerID="0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1"
Nov 03 01:17:09.087483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.087406    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1"} err="failed to get container status \"0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1\": rpc error: code = NotFound desc = could not find container \"0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1\": container with ID starting with 0585e54833e0f8db23a75afa5066812521f985c4abbade72da4c0b34f6eb80b1 not found: ID does not exist"
Nov 03 01:17:09.087483 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.087445    4550 scope.go:122] "RemoveContainer" containerID="38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4"
Nov 03 01:17:09.097272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.097248    4550 scope.go:122] "RemoveContainer" containerID="38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4"
Nov 03 01:17:09.097599 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:09.097569    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4\": container with ID starting with 38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4 not found: ID does not exist" containerID="38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4"
Nov 03 01:17:09.097712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.097608    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4"} err="failed to get container status \"38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4\": rpc error: code = NotFound desc = could not find container \"38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4\": container with ID starting with 38efaa45a7cf6003c11bdec749683555ebe499c497aece9e9564464b48ea40d4 not found: ID does not exist"
Nov 03 01:17:09.162790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.162754    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"]
Nov 03 01:17:09.176514 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.176481    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-2-w4-99bdb8-kserve-849c5d68d4-8mgth"]
Nov 03 01:17:09.203534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.203495    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"]
Nov 03 01:17:09.206225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.206189    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/simulator-w4-99bdb8-kserve-8494847845-mxsfz"]
Nov 03 01:17:09.240964 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.240926    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"]
Nov 03 01:17:09.246356 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.246327    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/distinct-w4-99bdb8-kserve-bfc6bf467-gklmv"]
Nov 03 01:17:09.282594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.282562    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"]
Nov 03 01:17:09.287359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.287330    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-99bdb8/premium-w4-99bdb8-kserve-54cdd7798d-cl4kj"]
Nov 03 01:17:09.808353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.808316    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17326228-95bb-449c-ac56-919975c8b94d" path="/var/lib/kubelet/pods/17326228-95bb-449c-ac56-919975c8b94d/volumes"
Nov 03 01:17:09.808826 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.808810    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" path="/var/lib/kubelet/pods/458c7292-7eae-491a-a1c1-4a4bbabe4ea1/volumes"
Nov 03 01:17:09.809223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.809209    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70820b0-0954-433d-a823-e8997a45a058" path="/var/lib/kubelet/pods/c70820b0-0954-433d-a823-e8997a45a058/volumes"
Nov 03 01:17:09.809955 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:09.809940    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" path="/var/lib/kubelet/pods/c8a89531-cdc1-4e23-bf99-3235a955c60f/volumes"
Nov 03 01:17:10.041355 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:10.041317    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"]
Nov 03 01:17:10.053894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:10.053860    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c4a9749-bxcck"]
Nov 03 01:17:10.532414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:10.532375    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:10.636678 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:10.636644    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:10.739464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:10.737095    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:11.809136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:11.809100    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f6a52f-4ab3-4b77-bfe2-dfbe45903309" path="/var/lib/kubelet/pods/44f6a52f-4ab3-4b77-bfe2-dfbe45903309/volumes"
Nov 03 01:17:12.014030 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:12.013988    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:12.184920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:12.184882    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:12.975803 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:12.975765    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:13.469863 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:13.469816    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:13.578230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:13.578194    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:16.342427 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:16.342389    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:18.154894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:18.154854    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:19.585757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.585706    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main" containerID="cri-o://538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84" gracePeriod=45
Nov 03 01:17:19.838500 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.838424    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 01:17:19.926921 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.926879    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home\") pod \"8919c830-d611-47e6-91c9-54e3d764d90e\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") "
Nov 03 01:17:19.926921 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.926918    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir\") pod \"8919c830-d611-47e6-91c9-54e3d764d90e\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") "
Nov 03 01:17:19.927178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.926999    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs\" (UniqueName: \"kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs\") pod \"8919c830-d611-47e6-91c9-54e3d764d90e\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") "
Nov 03 01:17:19.927178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927044    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm\") pod \"8919c830-d611-47e6-91c9-54e3d764d90e\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") "
Nov 03 01:17:19.927178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927126    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg\" (UniqueName: \"kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg\") pod \"8919c830-d611-47e6-91c9-54e3d764d90e\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") "
Nov 03 01:17:19.927178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927159    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache\") pod \"8919c830-d611-47e6-91c9-54e3d764d90e\" (UID: \"8919c830-d611-47e6-91c9-54e3d764d90e\") "
Nov 03 01:17:19.927381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927271    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir" pod "8919c830-d611-47e6-91c9-54e3d764d90e" (UID: "8919c830-d611-47e6-91c9-54e3d764d90e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.927381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927289    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home" pod "8919c830-d611-47e6-91c9-54e3d764d90e" (UID: "8919c830-d611-47e6-91c9-54e3d764d90e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.927552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927530    4550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-home\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.927608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927586    4550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-tmp-dir\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.927712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.927668    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache" pod "8919c830-d611-47e6-91c9-54e3d764d90e" (UID: "8919c830-d611-47e6-91c9-54e3d764d90e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.929491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.929455    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs" pod "8919c830-d611-47e6-91c9-54e3d764d90e" (UID: "8919c830-d611-47e6-91c9-54e3d764d90e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:19.929674 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.929656    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm" pod "8919c830-d611-47e6-91c9-54e3d764d90e" (UID: "8919c830-d611-47e6-91c9-54e3d764d90e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.929975 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:19.929958    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg" pod "8919c830-d611-47e6-91c9-54e3d764d90e" (UID: "8919c830-d611-47e6-91c9-54e3d764d90e"). InnerVolumeSpecName "kube-api-access-b49jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:20.026198 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.026166    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"]
Nov 03 01:17:20.028357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.028330    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cedfe99f-fv4qc"]
Nov 03 01:17:20.028498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.028473    4550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8919c830-d611-47e6-91c9-54e3d764d90e-tls-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.028498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.028493    4550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-dshm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.028595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.028503    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b49jg\" (UniqueName: \"kubernetes.io/projected/8919c830-d611-47e6-91c9-54e3d764d90e-kube-api-access-b49jg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.028595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.028514    4550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8919c830-d611-47e6-91c9-54e3d764d90e-model-cache\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.123238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.123153    4550 generic.go:356] "Generic (PLEG): container finished" podID="8919c830-d611-47e6-91c9-54e3d764d90e" containerID="538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84" exitCode=0
Nov 03 01:17:20.123238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.123207    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"
Nov 03 01:17:20.123453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.123241    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" event={"ID":"8919c830-d611-47e6-91c9-54e3d764d90e","Type":"ContainerDied","Data":"538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84"}
Nov 03 01:17:20.123453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.123300    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8" event={"ID":"8919c830-d611-47e6-91c9-54e3d764d90e","Type":"ContainerDied","Data":"b24e316df3791e7f51c77246dab1df0ce4cb94382813e776cf8019cbb0647f82"}
Nov 03 01:17:20.123453 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.123324    4550 scope.go:122] "RemoveContainer" containerID="538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84"
Nov 03 01:17:20.133982 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.133959    4550 scope.go:122] "RemoveContainer" containerID="538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84"
Nov 03 01:17:20.134276 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:20.134259    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84\": container with ID starting with 538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84 not found: ID does not exist" containerID="538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84"
Nov 03 01:17:20.134362 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.134285    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84"} err="failed to get container status \"538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84\": rpc error: code = NotFound desc = could not find container \"538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84\": container with ID starting with 538cd6263bdcf05714518c59a29233128f34d838a80fba28df13525313eaeb84 not found: ID does not exist"
Nov 03 01:17:20.176793 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.176764    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"]
Nov 03 01:17:20.182682 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:20.182656    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/simulator-w1-b85343-kserve-5f875cc989-98rt8"]
Nov 03 01:17:21.806656 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:21.806622    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cdb7f84-b258-4727-bdac-b4c6a36a162f" path="/var/lib/kubelet/pods/5cdb7f84-b258-4727-bdac-b4c6a36a162f/volumes"
Nov 03 01:17:21.807063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:21.807038    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" path="/var/lib/kubelet/pods/8919c830-d611-47e6-91c9-54e3d764d90e/volumes"
Nov 03 01:17:23.158367 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:23.158338    4550 scope.go:122] "RemoveContainer" containerID="265620a0affcbf536fa8d2ffd2a352347e67c9178bd34a6e7a8c650bf2c532d5"
Nov 03 01:17:23.168716 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:23.168692    4550 scope.go:122] "RemoveContainer" containerID="3ac5a7589801c2b195a73fd78baaf5fd91deb521ed5edfc17ee7e18db8869f0e"
Nov 03 01:17:24.037786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:24.037748    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:24.377768 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:24.377678    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:24.724484 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:24.724442    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:17:27.991103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991058    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"]
Nov 03 01:17:27.991673 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991652    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main"
Nov 03 01:17:27.991726 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991678    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main"
Nov 03 01:17:27.991726 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991701    4550 cpu_manager.go:414] "containerMap: removing container" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main"
Nov 03 01:17:27.991726 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991708    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991726    4550 cpu_manager.go:414] "containerMap: removing container" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991735    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991749    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991756    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991779    4550 cpu_manager.go:414] "containerMap: removing container" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991787    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991802    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main"
Nov 03 01:17:27.991824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991810    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991830    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b3eb199b-763b-44a6-99d4-2074beac9367" containerName="revoke-keys"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991838    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3eb199b-763b-44a6-99d4-2074beac9367" containerName="revoke-keys"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991941    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a89531-cdc1-4e23-bf99-3235a955c60f" containerName="main"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991957    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3eb199b-763b-44a6-99d4-2074beac9367" containerName="revoke-keys"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991968    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8919c830-d611-47e6-91c9-54e3d764d90e" containerName="main"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991981    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="17326228-95bb-449c-ac56-919975c8b94d" containerName="main"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.991991    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="458c7292-7eae-491a-a1c1-4a4bbabe4ea1" containerName="main"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.992000    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70820b0-0954-433d-a823-e8997a45a058" containerName="main"
Nov 03 01:17:27.992125 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.992011    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="424b73b2-0601-44b7-8aa2-20734d0aa15a" containerName="istio-proxy"
Nov 03 01:17:27.995585 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:27.995563    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.021486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.017294    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"]
Nov 03 01:17:28.099053 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.099022    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-99bdb8-openshift-default-dockercfg-dq2f2\""
Nov 03 01:17:28.107504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107473    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107533    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107616    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107636    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107854 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107665    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz77s\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107854 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107758    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107854 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107781    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.107854 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107821    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.108082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.107924    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209264    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209334    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209365    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209399    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz77s\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209462    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209495    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209557 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209546    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209573    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.209942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209631    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.210073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.209995    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.210185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.210157    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.210792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.210741    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.211329 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.211306    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.211727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.211673    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.213456 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.213337    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.213613 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.213590    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.229695 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.229673    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz77s\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.230047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.230026    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.317897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.317807    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:28.632170 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:17:28.632136    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c196e57_4420_4397_8a08_b22399ed3825.slice/crio-61a9bd620ab2162df7432b902e9d68b2f316482c929650b77385f15fef1aee5c WatchSource:0}: Error finding container 61a9bd620ab2162df7432b902e9d68b2f316482c929650b77385f15fef1aee5c: Status 404 returned error can't find the container with id 61a9bd620ab2162df7432b902e9d68b2f316482c929650b77385f15fef1aee5c
Nov 03 01:17:28.632926 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.632898    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"]
Nov 03 01:17:28.634195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.634166    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:17:28.634291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.634233    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:17:28.634291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:28.634261    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:17:29.186291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:29.186252    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" event={"ID":"0c196e57-4420-4397-8a08-b22399ed3825","Type":"ContainerStarted","Data":"760d7b6bfb4b2415bd4bc2d42b64b77654fcf722aadd389c59a556a5950e3b9f"}
Nov 03 01:17:29.186931 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:29.186906    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" event={"ID":"0c196e57-4420-4397-8a08-b22399ed3825","Type":"ContainerStarted","Data":"61a9bd620ab2162df7432b902e9d68b2f316482c929650b77385f15fef1aee5c"}
Nov 03 01:17:29.245825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:29.245773    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" podStartSLOduration=2.24576047 podStartE2EDuration="2.24576047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:29.243600244 +0000 UTC m=+4331.565137063" watchObservedRunningTime="2026-11-03 01:17:29.24576047 +0000 UTC m=+4331.567297270"
Nov 03 01:17:29.319009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:29.318961    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:29.320700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:29.320671    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 01:17:29.320846 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:29.320722    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 01:17:30.319177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:30.319138    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 01:17:30.319704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:30.319198    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 01:17:30.329836 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:30.329773    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:30.615306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:30.615212    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:30.995387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:30.995274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:31.237087 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:31.237053    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:31.318805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:31.318693    4550 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused" start-of-body=
Nov 03 01:17:31.318805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:31.318761    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.58:15021/healthz/ready\": dial tcp 10.133.1.58:15021: connect: connection refused"
Nov 03 01:17:31.350372 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:31.350266    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:31.515042 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:31.515008    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:31.692890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:31.692842    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:32.093556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:32.090175    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"]
Nov 03 01:17:32.113744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:32.113712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:32.213618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:32.213582    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy" containerID="cri-o://760d7b6bfb4b2415bd4bc2d42b64b77654fcf722aadd389c59a556a5950e3b9f" gracePeriod=30
Nov 03 01:17:36.970558 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:36.970521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:37.079228 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:37.079178    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:40.256169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.256132    4550 generic.go:356] "Generic (PLEG): container finished" podID="0c196e57-4420-4397-8a08-b22399ed3825" containerID="760d7b6bfb4b2415bd4bc2d42b64b77654fcf722aadd389c59a556a5950e3b9f" exitCode=0
Nov 03 01:17:40.256539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.256177    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" event={"ID":"0c196e57-4420-4397-8a08-b22399ed3825","Type":"ContainerDied","Data":"760d7b6bfb4b2415bd4bc2d42b64b77654fcf722aadd389c59a556a5950e3b9f"}
Nov 03 01:17:40.256539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.256239    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764" event={"ID":"0c196e57-4420-4397-8a08-b22399ed3825","Type":"ContainerDied","Data":"61a9bd620ab2162df7432b902e9d68b2f316482c929650b77385f15fef1aee5c"}
Nov 03 01:17:40.256539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.256257    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61a9bd620ab2162df7432b902e9d68b2f316482c929650b77385f15fef1aee5c"
Nov 03 01:17:40.265441 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.265408    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:40.356649 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356572    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.356649 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356634    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.356871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356676    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.356871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356707    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.356871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356777    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.356871 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356827    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.357035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356894    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.357035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356968    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.357035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.356999    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s\") pod \"0c196e57-4420-4397-8a08-b22399ed3825\" (UID: \"0c196e57-4420-4397-8a08-b22399ed3825\") "
Nov 03 01:17:40.358804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.358738    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.358942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.358842    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:40.359020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.358948    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.359020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.358976    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:40.359146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.359100    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.359188 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.359143    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.360746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.360719    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:40.361280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.361263    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "kube-api-access-gz77s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:40.361280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.361269    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy" pod "0c196e57-4420-4397-8a08-b22399ed3825" (UID: "0c196e57-4420-4397-8a08-b22399ed3825"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.458223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458183    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458223    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz77s\" (UniqueName: \"kubernetes.io/projected/0c196e57-4420-4397-8a08-b22399ed3825-kube-api-access-gz77s\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458237    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c196e57-4420-4397-8a08-b22399ed3825-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458250    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458265    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c196e57-4420-4397-8a08-b22399ed3825-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458277    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458290    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458302    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.458420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:40.458321    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c196e57-4420-4397-8a08-b22399ed3825-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:41.262662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:41.262624    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"
Nov 03 01:17:41.337529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:41.337487    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"]
Nov 03 01:17:41.341600 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:41.341169    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-jb764"]
Nov 03 01:17:41.341833 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:41.341721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:41.442040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:41.442012    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:41.806178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:41.806140    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c196e57-4420-4397-8a08-b22399ed3825" path="/var/lib/kubelet/pods/0c196e57-4420-4397-8a08-b22399ed3825/volumes"
Nov 03 01:17:42.759363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:42.759324    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:43.436297 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:43.436261    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:43.534589 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:43.534549    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:44.796036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:44.796003    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:45.142662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:45.142575    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"]
Nov 03 01:17:45.142904 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:45.142853    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy" containerID="cri-o://8eaec5d73144fe1791a459a908bd2affcce88a4d84fbc607d182d0dd439c6e0d" gracePeriod=30
Nov 03 01:17:45.176299 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:45.176253    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:46.869009 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:46.868979    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jk4jg_624eee0c-d8af-4382-876c-f8ddac59b3cf/kuadrant-console-plugin/0.log"
Nov 03 01:17:47.520371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:47.520334    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-89stt_0b1877da-8248-4ab3-b721-c154ffa7e09f/manager/0.log"
Nov 03 01:17:48.092198 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:48.092157    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:48.379642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:48.379556    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:48.480382 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:48.480346    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:17:49.408083 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:49.408049    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2hwfp_8588e986-19d2-4ce9-9c19-fc2ae2859e26/manager/0.log"
Nov 03 01:17:49.627957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:49.627919    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lrpvq_5a235b0f-ffbd-4cb2-bc29-19ade07bb186/manager/0.log"
Nov 03 01:17:49.849152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:49.849072    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p876r_1de0029a-6ab4-47db-9562-805aab89a6ea/manager/0.log"
Nov 03 01:17:50.026688 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.026655    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"]
Nov 03 01:17:50.039244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.039204    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ef8ba0-l4kv7"]
Nov 03 01:17:50.313626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.313579    4550 generic.go:356] "Generic (PLEG): container finished" podID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerID="8eaec5d73144fe1791a459a908bd2affcce88a4d84fbc607d182d0dd439c6e0d" exitCode=0
Nov 03 01:17:50.313822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.313660    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" event={"ID":"e8a5789a-6d7c-43a4-916d-c554555dbb04","Type":"ContainerDied","Data":"8eaec5d73144fe1791a459a908bd2affcce88a4d84fbc607d182d0dd439c6e0d"}
Nov 03 01:17:50.398413 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.398386    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 01:17:50.557759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557671    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.557759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557762    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557806    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557860    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557908    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.557949    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558033    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558062    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs\") pod \"e8a5789a-6d7c-43a4-916d-c554555dbb04\" (UID: \"e8a5789a-6d7c-43a4-916d-c554555dbb04\") "
Nov 03 01:17:50.558284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558130    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:50.558712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558329    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:50.558712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558394    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a5789a-6d7c-43a4-916d-c554555dbb04-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.558712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558427    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:50.558712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558505    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:50.558712 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.558580    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:50.560327 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.560293    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:50.560327 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.560308    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "kube-api-access-kbg74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:50.560558 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.560540    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:50.560834 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.560806    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token" pod "e8a5789a-6d7c-43a4-916d-c554555dbb04" (UID: "e8a5789a-6d7c-43a4-916d-c554555dbb04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:50.561370 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.561354    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hfkg4_4c24118b-7393-4ad0-96ae-11d71d2dad5b/server/0.log"
Nov 03 01:17:50.659787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659746    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.659787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659779    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbg74\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-kube-api-access-kbg74\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.659787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659788    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.659787 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659799    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.660103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659808    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.660103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659817    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.660103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659825    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a5789a-6d7c-43a4-916d-c554555dbb04-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:50.660103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:50.659833    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a5789a-6d7c-43a4-916d-c554555dbb04-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:17:51.321947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.321919    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"
Nov 03 01:17:51.321947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.321940    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6" event={"ID":"e8a5789a-6d7c-43a4-916d-c554555dbb04","Type":"ContainerDied","Data":"88e8acfff6195d81584635de12673ad182d866a6ff442c93cd29e7d4160fac2c"}
Nov 03 01:17:51.322224 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.321997    4550 scope.go:122] "RemoveContainer" containerID="8eaec5d73144fe1791a459a908bd2affcce88a4d84fbc607d182d0dd439c6e0d"
Nov 03 01:17:51.376092 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.376057    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"]
Nov 03 01:17:51.389250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.389210    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-sv8v6"]
Nov 03 01:17:51.805881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.805843    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" path="/var/lib/kubelet/pods/e8a5789a-6d7c-43a4-916d-c554555dbb04/volumes"
Nov 03 01:17:51.806283 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:17:51.806242    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f653e3-9e68-428e-b7dc-2b74151e6b75" path="/var/lib/kubelet/pods/f6f653e3-9e68-428e-b7dc-2b74151e6b75/volumes"
Nov 03 01:17:54.722507 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:17:54.722460    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:18:00.334805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:00.334721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:00.802675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:00.802642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:00.956582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:00.956545    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:03.747525 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:03.747493    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:03.848980 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:03.848949    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:05.295822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:05.295783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:05.399443 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:05.399391    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:05.499763 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:05.499726    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:05.601586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:05.601506    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:06.034152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:06.034119    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"]
Nov 03 01:18:06.047318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:06.047286    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42dab15d-dd69l"]
Nov 03 01:18:07.119017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:07.118984    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:07.806480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:07.806427    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee38c88b-5480-49c0-a796-d4c37df8e133" path="/var/lib/kubelet/pods/ee38c88b-5480-49c0-a796-d4c37df8e133/volumes"
Nov 03 01:18:08.735423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:08.735385    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:10.659293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:10.659260    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:12.443075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:12.443043    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:13.349602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:13.349566    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"]
Nov 03 01:18:13.350071 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:13.349767    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy" containerID="cri-o://590cb62a9db3252f6d0eba46a77b3ea7bf76a71de2c834907bd6464df6275b76" gracePeriod=30
Nov 03 01:18:14.202599 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:14.202568    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:16.023950 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:16.023913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:16.468515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:16.468480    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:17.730804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:17.730738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:18.467665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.467634    4550 generic.go:356] "Generic (PLEG): container finished" podID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerID="590cb62a9db3252f6d0eba46a77b3ea7bf76a71de2c834907bd6464df6275b76" exitCode=0
Nov 03 01:18:18.467847 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.467705    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" event={"ID":"f8374e40-3606-493e-9c09-f7ebecf7a6af","Type":"ContainerDied","Data":"590cb62a9db3252f6d0eba46a77b3ea7bf76a71de2c834907bd6464df6275b76"}
Nov 03 01:18:18.598861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.598828    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:18:18.723881 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.723843    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.723900    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.723928    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.723969    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.723999    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724034    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724061    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.724115 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724111    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724217    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket\") pod \"f8374e40-3606-493e-9c09-f7ebecf7a6af\" (UID: \"f8374e40-3606-493e-9c09-f7ebecf7a6af\") "
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724270    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724532    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724567    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724578    4550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724656    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:18.725047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.724734    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:18.726490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.726466    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:18.726490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.726484    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "kube-api-access-qxx4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:18.726650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.726634    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:18.726691 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.726627    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo" pod "f8374e40-3606-493e-9c09-f7ebecf7a6af" (UID: "f8374e40-3606-493e-9c09-f7ebecf7a6af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:18:18.825775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825737    4550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-credential-socket\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.825775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825771    4550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-envoy\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.825775 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825780    4550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-workload-certs\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.826267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825787    4550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-data\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.826267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825797    4550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8374e40-3606-493e-9c09-f7ebecf7a6af-istiod-ca-cert\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.826267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825809    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxx4n\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-kube-api-access-qxx4n\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.826267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825818    4550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-podinfo\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:18.826267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:18.825827    4550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8374e40-3606-493e-9c09-f7ebecf7a6af-istio-token\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:18:19.428154 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.428112    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:19.477192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.477164    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"
Nov 03 01:18:19.477404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.477163    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q" event={"ID":"f8374e40-3606-493e-9c09-f7ebecf7a6af","Type":"ContainerDied","Data":"e65c08b409be039aca94194d4805abbd95684a2ce76bc06d7100191ef39d70f3"}
Nov 03 01:18:19.477404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.477293    4550 scope.go:122] "RemoveContainer" containerID="590cb62a9db3252f6d0eba46a77b3ea7bf76a71de2c834907bd6464df6275b76"
Nov 03 01:18:19.545190 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.545156    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"]
Nov 03 01:18:19.558612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.558568    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-mnd7q"]
Nov 03 01:18:19.806759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:19.806681    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" path="/var/lib/kubelet/pods/f8374e40-3606-493e-9c09-f7ebecf7a6af/volumes"
Nov 03 01:18:21.308779 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:21.308741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:23.140569 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:23.140529    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:23.231612 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:23.231576    4550 scope.go:122] "RemoveContainer" containerID="1827033eceaec0033be3cf437c968ad52d2158da98a7c9cc7b5a8baa004ee86b"
Nov 03 01:18:23.240100 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:23.240078    4550 scope.go:122] "RemoveContainer" containerID="f715ee685132970e1d94f4d153d22ecdff7809b161ac6aad8ab2068e0546e12f"
Nov 03 01:18:23.247594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:23.247576    4550 scope.go:122] "RemoveContainer" containerID="ddf55d0803314d0fcaafa46b22f197d653692bc99d140761786b661734a281a9"
Nov 03 01:18:23.255373 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:23.255353    4550 scope.go:122] "RemoveContainer" containerID="0d513897b2de99e3dbe5c40b2e278ace49b366361b1913048aa63fe05332e994"
Nov 03 01:18:24.722248 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:18:24.722214    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:18:24.845368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:24.845336    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:26.533805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:26.533770    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:28.124670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:28.124634    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:29.831360 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:29.831319    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:30.258512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:30.258475    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:30.365255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:30.365222    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:31.519967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:31.519931    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:32.904468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:32.904416    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:33.426461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:33.426414    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:33.527946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:33.527913    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:35.914739 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:35.914702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:36.027254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:36.027218    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:39.032233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:18:39.032197    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:18:54.721453 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:18:54.721400    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:19:12.317049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:12.317015    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:19:13.567681 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.567632    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"]
Nov 03 01:19:13.568205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568183    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy"
Nov 03 01:19:13.568279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568209    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy"
Nov 03 01:19:13.568279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568235    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy"
Nov 03 01:19:13.568279 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568244    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy"
Nov 03 01:19:13.568484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568285    4550 cpu_manager.go:414] "containerMap: removing container" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy"
Nov 03 01:19:13.568484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568293    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy"
Nov 03 01:19:13.568484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568389    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a5789a-6d7c-43a4-916d-c554555dbb04" containerName="istio-proxy"
Nov 03 01:19:13.568484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568404    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c196e57-4420-4397-8a08-b22399ed3825" containerName="istio-proxy"
Nov 03 01:19:13.568484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.568417    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8374e40-3606-493e-9c09-f7ebecf7a6af" containerName="istio-proxy"
Nov 03 01:19:13.572591 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.572567    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.589944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.589918    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"]
Nov 03 01:19:13.714928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.714898    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.714928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.714932    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.714962    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77lvl\" (UniqueName: \"kubernetes.io/projected/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-kube-api-access-77lvl\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.715031    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.715069    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.715090    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.715111    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.715140    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.715301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.715175    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816099 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816074    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816099 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816103    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816244    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816283    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816341    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77lvl\" (UniqueName: \"kubernetes.io/projected/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-kube-api-access-77lvl\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816475    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816486    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816529    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816561    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816586    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816668    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816734    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.816960 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.816937    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.817248 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.817225    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.818470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.818398    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.818850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.818832    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.835990 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.835965    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77lvl\" (UniqueName: \"kubernetes.io/projected/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-kube-api-access-77lvl\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.836081 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.835967    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-gknbx\" (UID: \"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:13.887104 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:13.887074    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:14.194973 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:19:14.194941    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89e0826_6db2_4e0d_8c06_e3e9ad3bdf71.slice/crio-f1ed5cd2febacaa4c7749b59c9aa221ee2b7e3b6dafc47bf314bc0e4db5126ac WatchSource:0}: Error finding container f1ed5cd2febacaa4c7749b59c9aa221ee2b7e3b6dafc47bf314bc0e4db5126ac: Status 404 returned error can't find the container with id f1ed5cd2febacaa4c7749b59c9aa221ee2b7e3b6dafc47bf314bc0e4db5126ac
Nov 03 01:19:14.195212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.195188    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"]
Nov 03 01:19:14.197107 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.197077    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:19:14.197196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.197140    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:19:14.197196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.197168    4550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 01:19:14.745686 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.745656    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:19:14.785541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.785504    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" event={"ID":"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71","Type":"ContainerStarted","Data":"a1e8cd40b77adea1d078df4978ff2e581a74fa91a7a54ab47a02877b31eea103"}
Nov 03 01:19:14.785723 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.785563    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" event={"ID":"f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71","Type":"ContainerStarted","Data":"f1ed5cd2febacaa4c7749b59c9aa221ee2b7e3b6dafc47bf314bc0e4db5126ac"}
Nov 03 01:19:14.842905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.842851    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podStartSLOduration=1.84283671 podStartE2EDuration="1.84283671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:14.841297562 +0000 UTC m=+4437.162834361" watchObservedRunningTime="2026-11-03 01:19:14.84283671 +0000 UTC m=+4437.164373558"
Nov 03 01:19:14.887217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:14.887178    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:15.888948 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:15.888915    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 01:19:15.889354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:15.888964    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 01:19:16.678845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:16.678759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:19:16.887816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:16.887673    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 01:19:16.887816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:16.887731    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 01:19:17.887404 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:17.887368    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 01:19:17.887920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:17.887421    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 01:19:18.887769 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:18.887734    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 01:19:18.888151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:18.887797    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 01:19:19.887957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:19.887914    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 01:19:19.888414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:19.887985    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 01:19:20.887864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:20.887823    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused" start-of-body=
Nov 03 01:19:20.888049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:20.887899    4550 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": dial tcp 10.133.1.59:15021: connect: connection refused"
Nov 03 01:19:22.214478 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:22.214419    4550 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:22.215044 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:22.215021    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:22.215837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:22.215818    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx"
Nov 03 01:19:23.294563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:23.294533    4550 scope.go:122] "RemoveContainer" containerID="dc6f576fc2204fb5041206db5e04da6ab76f7169c494d2760f2567aa21054d21"
Nov 03 01:19:23.303035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:23.303015    4550 scope.go:122] "RemoveContainer" containerID="ad4b65603776a8cef6ffa9fd7a9ecc40546321fd09f612c474c0c2ae38ffb832"
Nov 03 01:19:24.721284 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:19:24.721251    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:19:31.027941 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:31.027862    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"]
Nov 03 01:19:31.041173 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:31.041146    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022d7e0c-cz77s"]
Nov 03 01:19:31.805495 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:31.805464    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f" path="/var/lib/kubelet/pods/583e0c31-6e9f-44a0-b4d0-3a33f9d84d2f/volumes"
Nov 03 01:19:41.033924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:41.033880    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"]
Nov 03 01:19:41.047266 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:41.047236    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f7606b7-f4lcm"]
Nov 03 01:19:41.809256 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:41.809223    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f61bd1-aef2-4709-aa1d-d0e2d78614e1" path="/var/lib/kubelet/pods/f6f61bd1-aef2-4709-aa1d-d0e2d78614e1/volumes"
Nov 03 01:19:53.028848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:53.028802    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"]
Nov 03 01:19:53.031323 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:53.030541    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f58a3b7c-wq4nx"]
Nov 03 01:19:53.033368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:53.033342    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"]
Nov 03 01:19:53.035710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:53.035686    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b949a7a-xhdgg"]
Nov 03 01:19:53.808644 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:53.808604    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9821e4-845e-4b4a-91f0-e067bdb9d5ce" path="/var/lib/kubelet/pods/2f9821e4-845e-4b4a-91f0-e067bdb9d5ce/volumes"
Nov 03 01:19:53.809103 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:53.809082    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac5ef82-a949-4911-93e0-b3c5a84714b2" path="/var/lib/kubelet/pods/8ac5ef82-a949-4911-93e0-b3c5a84714b2/volumes"
Nov 03 01:19:54.727140 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:19:54.727103    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:19:57.568586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:57.568539    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:19:57.569167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:57.569135    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:19:59.026251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:59.026216    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"]
Nov 03 01:19:59.039367 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:59.039337    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c73b89-bmnfd"]
Nov 03 01:19:59.228806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:59.228769    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:19:59.806786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:19:59.806753    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65" path="/var/lib/kubelet/pods/1c23b1de-e9f8-4c2f-89cb-cc658f4d5f65/volumes"
Nov 03 01:20:00.558594 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:00.558557    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:20:03.040036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:03.040000    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"]
Nov 03 01:20:03.053045 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:03.053006    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5375652-wrdgl"]
Nov 03 01:20:03.853203 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:03.845030    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8778b9c3-cee6-42d9-8ccd-ed5ddc189c84" path="/var/lib/kubelet/pods/8778b9c3-cee6-42d9-8ccd-ed5ddc189c84/volumes"
Nov 03 01:20:04.038567 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:04.038527    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"]
Nov 03 01:20:04.049827 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:04.049787    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33808623-l9fd5"]
Nov 03 01:20:05.806330 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:05.806286    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015206dc-1fd7-4f34-a0b0-89e0255b1720" path="/var/lib/kubelet/pods/015206dc-1fd7-4f34-a0b0-89e0255b1720/volumes"
Nov 03 01:20:10.028986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:10.028946    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"]
Nov 03 01:20:10.042232 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:10.042199    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc51a9ae-rc95p"]
Nov 03 01:20:11.805891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:11.805860    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a14dcc-b531-485f-9a58-8c084ed312b1" path="/var/lib/kubelet/pods/56a14dcc-b531-485f-9a58-8c084ed312b1/volumes"
Nov 03 01:20:15.046285 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:15.046201    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"]
Nov 03 01:20:15.059320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:15.059280    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d1bb7b3-h4bzb"]
Nov 03 01:20:15.806484 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:15.806447    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0237abd9-bc38-46c7-b3f4-a8fa462d26d8" path="/var/lib/kubelet/pods/0237abd9-bc38-46c7-b3f4-a8fa462d26d8/volumes"
Nov 03 01:20:16.676378 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:16.676335    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:20:18.557987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:18.557934    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:20:21.036672 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:21.036609    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:20:21.037221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:21.036721    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:20:24.723919 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:20:24.723869    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:20:25.028861 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:25.028750    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"]
Nov 03 01:20:25.041917 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:25.041877    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fc69f12-74s4l"]
Nov 03 01:20:25.812731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:25.812693    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec57530-5efd-44f2-84bc-7cc817e99c2c" path="/var/lib/kubelet/pods/0ec57530-5efd-44f2-84bc-7cc817e99c2c/volumes"
Nov 03 01:20:46.586024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:46.585978    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:20:46.927393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:46.927351    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:20:48.387460 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:48.386416    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:20:53.262604 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:53.262555    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:20:53.263858 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:20:53.263815    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:20:54.723793 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:20:54.723750    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:21:16.798911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:16.798873    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:21:23.336445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.336387    4550 scope.go:122] "RemoveContainer" containerID="6ca3a602e5b20c0b17fd4ae98d39bb78b9623322218ff049756af2bc8bbe308d"
Nov 03 01:21:23.345015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.344990    4550 scope.go:122] "RemoveContainer" containerID="de78f22148ed96800fe63708c2592d3637d3de0129c46591f54312bcb756702e"
Nov 03 01:21:23.353017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.352993    4550 scope.go:122] "RemoveContainer" containerID="dccd77d2983fac6b423a927381ce635d398668aa60f9651793563255e16b0de2"
Nov 03 01:21:23.360733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.360711    4550 scope.go:122] "RemoveContainer" containerID="74702cd8a2498a34b98b72a729deaaf5de682643a60e84eaf69bd360d0605af5"
Nov 03 01:21:23.368570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.368552    4550 scope.go:122] "RemoveContainer" containerID="63e024f2ed3fdf0a0c71c788cf2890f073897d9f5e9fb736a300cf11e9f3694e"
Nov 03 01:21:23.377178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.377160    4550 scope.go:122] "RemoveContainer" containerID="7d7e74d161507aead60b7b5d359e139abd06e38f51448cd28f409d9bd46a5c5f"
Nov 03 01:21:23.385067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.385049    4550 scope.go:122] "RemoveContainer" containerID="279ac245be4752bdb3ae2b810e2e4a1454c6dc21332bff3b2ca6c8b5e13b0adf"
Nov 03 01:21:23.392184 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.392167    4550 scope.go:122] "RemoveContainer" containerID="8c454f76c6eefbdbeb6170b0c451c4960929586287d387aad0b8958f1eb592d6"
Nov 03 01:21:23.399354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.399336    4550 scope.go:122] "RemoveContainer" containerID="2c5db9d9aeaed15af9b47a48e6bc06526e122c239b40d39b116db7b22b548ff8"
Nov 03 01:21:23.406522 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:23.406492    4550 scope.go:122] "RemoveContainer" containerID="31c008f3511fd6eb7699e33be992887fd143080e714847b812d12929a6857886"
Nov 03 01:21:24.722158 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:21:24.722122    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:21:49.503728 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:49.503680    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:21:49.603062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:49.603029    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:21:51.323300 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:51.323263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:21:51.551152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:51.551104    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:21:52.028324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:52.028293    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"]
Nov 03 01:21:52.041203 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:52.041165    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed325440-rgg2v"]
Nov 03 01:21:53.259631 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:53.259586    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:21:53.260280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:53.260241    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:21:53.833931 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:53.833894    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a3f5e1-372a-479a-adbc-9be2176afb02" path="/var/lib/kubelet/pods/66a3f5e1-372a-479a-adbc-9be2176afb02/volumes"
Nov 03 01:21:54.723841 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:21:54.723802    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:21:57.533894 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:57.533843    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:21:57.534635 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:21:57.534592    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:22:04.026903 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:04.026861    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"]
Nov 03 01:22:04.039824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:04.039793    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd658052-5cvth"]
Nov 03 01:22:04.245151 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:04.245114    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:04.700581 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:04.700546    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:04.915011 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:04.914949    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:05.805979 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:05.805949    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3eb199b-763b-44a6-99d4-2074beac9367" path="/var/lib/kubelet/pods/b3eb199b-763b-44a6-99d4-2074beac9367/volumes"
Nov 03 01:22:08.492896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:08.492859    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:08.609759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:08.609723    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:10.077112 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.077077    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:10.174965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.174908    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:10.275912 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.275879    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:10.320376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.320340    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"]
Nov 03 01:22:10.324025 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.324008    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.348261 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.348228    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"]
Nov 03 01:22:10.357619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.357595    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-adf1237d-dockercfg-jdwtq\""
Nov 03 01:22:10.357619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.357616    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b9d7ac\""
Nov 03 01:22:10.377776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.377742    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"]
Nov 03 01:22:10.381209 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.381190    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.400834 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.400806    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"]
Nov 03 01:22:10.443093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.443060    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.443301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.443110    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.443301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.443146    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfnt\" (UniqueName: \"kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.443301 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.443179    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzt2\" (UniqueName: \"kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.543770 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.543727    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.543770 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.543777    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfnt\" (UniqueName: \"kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.544069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.543804    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzt2\" (UniqueName: \"kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.544069 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.543881    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.544548 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.544527    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.544632 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.544612    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.564629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.564605    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfnt\" (UniqueName: \"kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.564762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.564608    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzt2\" (UniqueName: \"kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.636703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.636623    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:10.694145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.693829    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:10.933354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.933325    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"]
Nov 03 01:22:10.933850 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:22:10.933816    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757fdfd9_8fbf_483b_bd63_a69f82309cf2.slice/crio-9187ce9328d407184b159969fbb0abf32abeb307f2acab7cf621f3dc34bfe29f WatchSource:0}: Error finding container 9187ce9328d407184b159969fbb0abf32abeb307f2acab7cf621f3dc34bfe29f: Status 404 returned error can't find the container with id 9187ce9328d407184b159969fbb0abf32abeb307f2acab7cf621f3dc34bfe29f
Nov 03 01:22:10.999048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:10.999020    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"]
Nov 03 01:22:10.999620 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:22:10.999592    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13c2127_231b_4b4c_a0cc_d6dd7cf80995.slice/crio-83372b2553d7fe5813e033c4dc35d4e98765e29d4a8f151ede2a0d64e43534cd WatchSource:0}: Error finding container 83372b2553d7fe5813e033c4dc35d4e98765e29d4a8f151ede2a0d64e43534cd: Status 404 returned error can't find the container with id 83372b2553d7fe5813e033c4dc35d4e98765e29d4a8f151ede2a0d64e43534cd
Nov 03 01:22:11.806647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.806611    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" event={"ID":"b13c2127-231b-4b4c-a0cc-d6dd7cf80995","Type":"ContainerStarted","Data":"d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d"}
Nov 03 01:22:11.807127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.806667    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" event={"ID":"b13c2127-231b-4b4c-a0cc-d6dd7cf80995","Type":"ContainerStarted","Data":"83372b2553d7fe5813e033c4dc35d4e98765e29d4a8f151ede2a0d64e43534cd"}
Nov 03 01:22:11.807127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.806692    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:11.808129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.808103    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" event={"ID":"757fdfd9-8fbf-483b-bd63-a69f82309cf2","Type":"ContainerStarted","Data":"4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc"}
Nov 03 01:22:11.808254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.808150    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" event={"ID":"757fdfd9-8fbf-483b-bd63-a69f82309cf2","Type":"ContainerStarted","Data":"9187ce9328d407184b159969fbb0abf32abeb307f2acab7cf621f3dc34bfe29f"}
Nov 03 01:22:11.808254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.808193    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:11.846842 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.846787    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" podStartSLOduration=1.846773762 podStartE2EDuration="1.846773762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:11.845019226 +0000 UTC m=+4614.166556027" watchObservedRunningTime="2026-11-03 01:22:11.846773762 +0000 UTC m=+4614.168310562"
Nov 03 01:22:11.883882 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.883831    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" podStartSLOduration=1.883815517 podStartE2EDuration="1.883815517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:11.883773453 +0000 UTC m=+4614.205310252" watchObservedRunningTime="2026-11-03 01:22:11.883815517 +0000 UTC m=+4614.205352316"
Nov 03 01:22:11.924721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:11.924689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:13.639423 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:13.639383    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:15.426699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:15.426660    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:16.919410 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:16.919370    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:17.127062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:17.127023    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:18.921690 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:18.921648    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:20.732398 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:20.732356    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:22.418450 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:22.418384    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:22.815288 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:22.815197    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:22:22.815470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:22.815320    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:22:23.340550 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:23.340514    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:23.947876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:23.947836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:24.051450 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:24.051402    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:24.151549 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:24.151508    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:24.329280 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:24.329200    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:24.723993 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:22:24.723927    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:22:25.564293 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:25.564256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:25.964686 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:25.964642    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:27.634450 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:27.634394    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:29.337559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:29.337521    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:31.133874 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:31.133836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:32.826208 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:32.826166    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:34.534338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:34.534299    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:36.228128 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:36.228081    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:37.930779 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:37.930738    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:39.627804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:39.627771    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:41.322666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:41.322632    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:43.077445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:43.077401    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:44.855277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:44.855211    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:46.530754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:46.530721    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:48.341454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:48.341404    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:49.948821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:49.948783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:51.649006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:51.648976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:53.330470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:53.330411    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:54.722784 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:22:54.722745    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:22:55.039541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:55.039460    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:56.179845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.179812    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"]
Nov 03 01:22:56.183720 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.183667    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.201416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.201386    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"]
Nov 03 01:22:56.212485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.212456    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:22:56.212677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.212585    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:22:56.280988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.280948    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.281194 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.281018    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.281194 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.281150    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vs62\" (UniqueName: \"kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.381782 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.381737    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.381974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.381845    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vs62\" (UniqueName: \"kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.381974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.381912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.382731 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.382690    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.384244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.384211    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.401963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.401935    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vs62\" (UniqueName: \"kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62\") pod \"maas-api-revoke-sub-7ea7e196-rmr8v\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.497310 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.497229    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:56.794702 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.794663    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"]
Nov 03 01:22:56.795469 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:22:56.795442    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24f226b_22f4_4304_81f8_9dab4fd1c9cc.slice/crio-269f656b92ff0112e64e59d605dab3d33c5fcf8f28b64d495e8beba552e08d10 WatchSource:0}: Error finding container 269f656b92ff0112e64e59d605dab3d33c5fcf8f28b64d495e8beba552e08d10: Status 404 returned error can't find the container with id 269f656b92ff0112e64e59d605dab3d33c5fcf8f28b64d495e8beba552e08d10
Nov 03 01:22:56.814215 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:56.814183    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:57.027665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:57.027628    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v" event={"ID":"c24f226b-22f4-4304-81f8-9dab4fd1c9cc","Type":"ContainerStarted","Data":"269f656b92ff0112e64e59d605dab3d33c5fcf8f28b64d495e8beba552e08d10"}
Nov 03 01:22:58.034120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:58.034086    4550 generic.go:356] "Generic (PLEG): container finished" podID="c24f226b-22f4-4304-81f8-9dab4fd1c9cc" containerID="508909d14fdd80ce042c35fb6c7b0b935f772b4e6fd71a100b80d93e6bdf8c38" exitCode=0
Nov 03 01:22:58.034564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:58.034173    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v" event={"ID":"c24f226b-22f4-4304-81f8-9dab4fd1c9cc","Type":"ContainerDied","Data":"508909d14fdd80ce042c35fb6c7b0b935f772b4e6fd71a100b80d93e6bdf8c38"}
Nov 03 01:22:58.521519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:58.521484    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:22:59.173544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.173513    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:22:59.307660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.307569    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62\" (UniqueName: \"kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62\") pod \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") "
Nov 03 01:22:59.307660 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.307623    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca\") pod \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") "
Nov 03 01:22:59.307938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.307721    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca\") pod \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\" (UID: \"c24f226b-22f4-4304-81f8-9dab4fd1c9cc\") "
Nov 03 01:22:59.308062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.308038    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca" pod "c24f226b-22f4-4304-81f8-9dab4fd1c9cc" (UID: "c24f226b-22f4-4304-81f8-9dab4fd1c9cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:22:59.309812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.309789    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca" pod "c24f226b-22f4-4304-81f8-9dab4fd1c9cc" (UID: "c24f226b-22f4-4304-81f8-9dab4fd1c9cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:22:59.309941 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.309815    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62" pod "c24f226b-22f4-4304-81f8-9dab4fd1c9cc" (UID: "c24f226b-22f4-4304-81f8-9dab4fd1c9cc"). InnerVolumeSpecName "kube-api-access-5vs62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:22:59.409095 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.409054    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vs62\" (UniqueName: \"kubernetes.io/projected/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-kube-api-access-5vs62\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:22:59.409095 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.409087    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:22:59.409095 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:22:59.409101    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f226b-22f4-4304-81f8-9dab4fd1c9cc-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:00.045758 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:00.045726    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"
Nov 03 01:23:00.045944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:00.045753    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v" event={"ID":"c24f226b-22f4-4304-81f8-9dab4fd1c9cc","Type":"ContainerDied","Data":"269f656b92ff0112e64e59d605dab3d33c5fcf8f28b64d495e8beba552e08d10"}
Nov 03 01:23:00.045944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:00.045802    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="269f656b92ff0112e64e59d605dab3d33c5fcf8f28b64d495e8beba552e08d10"
Nov 03 01:23:00.225259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:00.225220    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:01.919683 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:01.919645    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:03.631147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:03.631102    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:05.441655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:05.441612    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:07.063972 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:07.063937    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:07.164000 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:07.163963    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:07.552319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:07.552285    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:08.919057 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:08.919019    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:10.552671 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:10.552639    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:12.124988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:12.124946    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:12.236791 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:12.236757    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:13.211992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:13.211952    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"]
Nov 03 01:23:13.242424 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:13.242390    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"]
Nov 03 01:23:13.348651 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:13.348610    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:13.492213 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:13.492113    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:13.773582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:13.773494    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:13.982474 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:13.982441    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:15.506470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:15.506390    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:17.036743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:17.036695    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:17.201472 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:17.201420    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:18.212337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:18.212266    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerName="payload-processing" containerID="cri-o://4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc" gracePeriod=25
Nov 03 01:23:18.243097 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:18.243047    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerName="payload-pre-processing" containerID="cri-o://d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d" gracePeriod=25
Nov 03 01:23:19.371608 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:19.371571    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:19.471810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:19.471767    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:22.813754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:22.813709    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:23:22.814195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:22.813724    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:23:23.463029 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:23.462994    4550 scope.go:122] "RemoveContainer" containerID="2d0e9efec3c8500e41411c418b2bbb7596b0b6752e56337b5ca5da58fbcaad20"
Nov 03 01:23:23.471737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:23.471715    4550 scope.go:122] "RemoveContainer" containerID="54ddde3bd11a79ced4b2adab5bad1ed2e98ce0cdcaf44536067c20907aa674ea"
Nov 03 01:23:24.723115 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:23:24.723080    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:23:24.805222 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:24.805188    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:32.813968 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:32.813927    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:23:32.814512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:32.813927    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:23:33.778849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:33.778809    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:38.491218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.491185    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:23:38.523776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.523747    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:23:38.582640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.582599    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume\") pod \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") "
Nov 03 01:23:38.582831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.582670    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2\" (UniqueName: \"kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2\") pod \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\" (UID: \"757fdfd9-8fbf-483b-bd63-a69f82309cf2\") "
Nov 03 01:23:38.582831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.582779    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt\" (UniqueName: \"kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt\") pod \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") "
Nov 03 01:23:38.582920 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.582828    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume\") pod \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\" (UID: \"b13c2127-231b-4b4c-a0cc-d6dd7cf80995\") "
Nov 03 01:23:38.583088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.583060    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume" pod "757fdfd9-8fbf-483b-bd63-a69f82309cf2" (UID: "757fdfd9-8fbf-483b-bd63-a69f82309cf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:23:38.583286 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.583264    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume" pod "b13c2127-231b-4b4c-a0cc-d6dd7cf80995" (UID: "b13c2127-231b-4b4c-a0cc-d6dd7cf80995"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:23:38.584913 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.584889    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt" pod "b13c2127-231b-4b4c-a0cc-d6dd7cf80995" (UID: "b13c2127-231b-4b4c-a0cc-d6dd7cf80995"). InnerVolumeSpecName "kube-api-access-5wfnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:38.584974 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.584927    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2" pod "757fdfd9-8fbf-483b-bd63-a69f82309cf2" (UID: "757fdfd9-8fbf-483b-bd63-a69f82309cf2"). InnerVolumeSpecName "kube-api-access-zzzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:38.684229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.684180    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757fdfd9-8fbf-483b-bd63-a69f82309cf2-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:38.684229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.684215    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzzt2\" (UniqueName: \"kubernetes.io/projected/757fdfd9-8fbf-483b-bd63-a69f82309cf2-kube-api-access-zzzt2\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:38.684229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.684225    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfnt\" (UniqueName: \"kubernetes.io/projected/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-kube-api-access-5wfnt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:38.684229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:38.684234    4550 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13c2127-231b-4b4c-a0cc-d6dd7cf80995-plugins-config-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:39.230320 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.230286    4550 generic.go:356] "Generic (PLEG): container finished" podID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerID="d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d" exitCode=0
Nov 03 01:23:39.230606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.230368    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"
Nov 03 01:23:39.230606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.230372    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" event={"ID":"b13c2127-231b-4b4c-a0cc-d6dd7cf80995","Type":"ContainerDied","Data":"d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d"}
Nov 03 01:23:39.230606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.230553    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76" event={"ID":"b13c2127-231b-4b4c-a0cc-d6dd7cf80995","Type":"ContainerDied","Data":"83372b2553d7fe5813e033c4dc35d4e98765e29d4a8f151ede2a0d64e43534cd"}
Nov 03 01:23:39.230606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.230584    4550 scope.go:122] "RemoveContainer" containerID="d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d"
Nov 03 01:23:39.232032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.232005    4550 generic.go:356] "Generic (PLEG): container finished" podID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerID="4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc" exitCode=0
Nov 03 01:23:39.232182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.232050    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" event={"ID":"757fdfd9-8fbf-483b-bd63-a69f82309cf2","Type":"ContainerDied","Data":"4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc"}
Nov 03 01:23:39.232182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.232068    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"
Nov 03 01:23:39.232182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.232088    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf" event={"ID":"757fdfd9-8fbf-483b-bd63-a69f82309cf2","Type":"ContainerDied","Data":"9187ce9328d407184b159969fbb0abf32abeb307f2acab7cf621f3dc34bfe29f"}
Nov 03 01:23:39.240306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.240286    4550 scope.go:122] "RemoveContainer" containerID="d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d"
Nov 03 01:23:39.240653 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:23:39.240631    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d\": container with ID starting with d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d not found: ID does not exist" containerID="d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d"
Nov 03 01:23:39.240747 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.240660    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d"} err="failed to get container status \"d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d\": rpc error: code = NotFound desc = could not find container \"d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d\": container with ID starting with d8a5649a312c1bf01dac10165b1c4aa26d2f2c8aeb27374fcdab755a16a6e91d not found: ID does not exist"
Nov 03 01:23:39.240747 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.240682    4550 scope.go:122] "RemoveContainer" containerID="4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc"
Nov 03 01:23:39.249097 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.249072    4550 scope.go:122] "RemoveContainer" containerID="4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc"
Nov 03 01:23:39.249401 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:23:39.249379    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc\": container with ID starting with 4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc not found: ID does not exist" containerID="4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc"
Nov 03 01:23:39.249480 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.249413    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc"} err="failed to get container status \"4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc\": rpc error: code = NotFound desc = could not find container \"4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc\": container with ID starting with 4bd11d6dc509b24c788a231f5bfd588b24c5edd8504d2af0b4e33ca588b604bc not found: ID does not exist"
Nov 03 01:23:39.282337 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.282294    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"]
Nov 03 01:23:39.295535 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.295502    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486d96cmf"]
Nov 03 01:23:39.330556 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.330520    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"]
Nov 03 01:23:39.331997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.331976    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8bhr76"]
Nov 03 01:23:39.805802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.805766    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" path="/var/lib/kubelet/pods/757fdfd9-8fbf-483b-bd63-a69f82309cf2/volumes"
Nov 03 01:23:39.806219 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:39.806148    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" path="/var/lib/kubelet/pods/b13c2127-231b-4b4c-a0cc-d6dd7cf80995/volumes"
Nov 03 01:23:44.378515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:44.378479    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:46.105343 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:46.105306    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:46.214021 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:46.213976    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:47.633776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:47.633741    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:47.753854 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:47.753811    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:47.854212 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:47.854177    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:49.446574 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:49.446536    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:49.598823 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:49.598783    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:50.080354 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:50.080316    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:51.137542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:51.137509    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:52.850401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:52.850365    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:54.522611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:54.522573    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:54.722038 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:23:54.722005    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:23:55.703817 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:55.703775    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:55.816957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:55.816924    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:56.092022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.091942    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"]
Nov 03 01:23:56.092333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092305    4550 cpu_manager.go:414] "containerMap: removing container" podUID="c24f226b-22f4-4304-81f8-9dab4fd1c9cc" containerName="revoke-keys"
Nov 03 01:23:56.092333 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092329    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24f226b-22f4-4304-81f8-9dab4fd1c9cc" containerName="revoke-keys"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092350    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerName="payload-pre-processing"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092360    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerName="payload-pre-processing"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092376    4550 cpu_manager.go:414] "containerMap: removing container" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerName="payload-processing"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092383    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerName="payload-processing"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092516    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24f226b-22f4-4304-81f8-9dab4fd1c9cc" containerName="revoke-keys"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092526    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13c2127-231b-4b4c-a0cc-d6dd7cf80995" containerName="payload-pre-processing"
Nov 03 01:23:56.092561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.092532    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="757fdfd9-8fbf-483b-bd63-a69f82309cf2" containerName="payload-processing"
Nov 03 01:23:56.097120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.097099    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.125528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.125505    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:23:56.125648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.125526    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:23:56.127231 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.127212    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"]
Nov 03 01:23:56.248809 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.248783    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzh46\" (UniqueName: \"kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.248967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.248824    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.248967 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.248857    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.330451 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.330396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:56.349369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.349308    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzh46\" (UniqueName: \"kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.349369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.349351    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.349542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.349379    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.350001 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.349977    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.351675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.351651    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.369130 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.369113    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzh46\" (UniqueName: \"kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46\") pod \"maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.409896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.409866    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:56.702914 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:56.702890    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"]
Nov 03 01:23:56.704630 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:23:56.704603    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59936056_800e_4bb0_a80e_40d9194a4d5e.slice/crio-a2a3fdc43d77e8729687962f2b2444f9cee5e2b8c2aec7327365e00a53f9cff0 WatchSource:0}: Error finding container a2a3fdc43d77e8729687962f2b2444f9cee5e2b8c2aec7327365e00a53f9cff0: Status 404 returned error can't find the container with id a2a3fdc43d77e8729687962f2b2444f9cee5e2b8c2aec7327365e00a53f9cff0
Nov 03 01:23:57.319951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:57.319916    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv" event={"ID":"59936056-800e-4bb0-a80e-40d9194a4d5e","Type":"ContainerStarted","Data":"f45e326e4c653efa790fb328b593cd789bdff2cb214a302c2e911435e8b25656"}
Nov 03 01:23:57.320129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:57.319974    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv" event={"ID":"59936056-800e-4bb0-a80e-40d9194a4d5e","Type":"ContainerStarted","Data":"a2a3fdc43d77e8729687962f2b2444f9cee5e2b8c2aec7327365e00a53f9cff0"}
Nov 03 01:23:57.359953 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:57.359909    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv" podStartSLOduration=1.359895484 podStartE2EDuration="1.359895484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:57.358702455 +0000 UTC m=+4719.680239254" watchObservedRunningTime="2026-11-03 01:23:57.359895484 +0000 UTC m=+4719.681432283"
Nov 03 01:23:58.038305 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:58.038274    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:23:58.325783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:58.325707    4550 generic.go:356] "Generic (PLEG): container finished" podID="59936056-800e-4bb0-a80e-40d9194a4d5e" containerID="f45e326e4c653efa790fb328b593cd789bdff2cb214a302c2e911435e8b25656" exitCode=0
Nov 03 01:23:58.325937 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:58.325794    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv" event={"ID":"59936056-800e-4bb0-a80e-40d9194a4d5e","Type":"ContainerDied","Data":"f45e326e4c653efa790fb328b593cd789bdff2cb214a302c2e911435e8b25656"}
Nov 03 01:23:59.469560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.469531    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:23:59.578559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.578517    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46\" (UniqueName: \"kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46\") pod \"59936056-800e-4bb0-a80e-40d9194a4d5e\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") "
Nov 03 01:23:59.578783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.578578    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca\") pod \"59936056-800e-4bb0-a80e-40d9194a4d5e\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") "
Nov 03 01:23:59.578783 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.578630    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca\") pod \"59936056-800e-4bb0-a80e-40d9194a4d5e\" (UID: \"59936056-800e-4bb0-a80e-40d9194a4d5e\") "
Nov 03 01:23:59.579062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.579026    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca" pod "59936056-800e-4bb0-a80e-40d9194a4d5e" (UID: "59936056-800e-4bb0-a80e-40d9194a4d5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:23:59.580903 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.580838    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca" pod "59936056-800e-4bb0-a80e-40d9194a4d5e" (UID: "59936056-800e-4bb0-a80e-40d9194a4d5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:23:59.580903 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.580865    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46" pod "59936056-800e-4bb0-a80e-40d9194a4d5e" (UID: "59936056-800e-4bb0-a80e-40d9194a4d5e"). InnerVolumeSpecName "kube-api-access-xzh46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:59.679537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.679499    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzh46\" (UniqueName: \"kubernetes.io/projected/59936056-800e-4bb0-a80e-40d9194a4d5e-kube-api-access-xzh46\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:59.679537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.679533    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:59.679537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.679544    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59936056-800e-4bb0-a80e-40d9194a4d5e-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:23:59.717205 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:23:59.717165    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:00.336387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:00.336349    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"
Nov 03 01:24:00.336387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:00.336376    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv" event={"ID":"59936056-800e-4bb0-a80e-40d9194a4d5e","Type":"ContainerDied","Data":"a2a3fdc43d77e8729687962f2b2444f9cee5e2b8c2aec7327365e00a53f9cff0"}
Nov 03 01:24:00.336647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:00.336459    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2a3fdc43d77e8729687962f2b2444f9cee5e2b8c2aec7327365e00a53f9cff0"
Nov 03 01:24:01.400724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:01.400688    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:03.029541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:03.029503    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:04.739987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:04.739943    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:06.440389 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:06.440350    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:06.626493 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:06.626452    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:06.725700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:06.725614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:07.930619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:07.930584    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:08.414729 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:08.414689    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:08.497047 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:08.497007    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:08.590748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:08.590712    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"]
Nov 03 01:24:08.605346 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:08.605302    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-b9d7ac-vr2gv"]
Nov 03 01:24:08.955174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:08.955135    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:09.805693 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:09.805657    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59936056-800e-4bb0-a80e-40d9194a4d5e" path="/var/lib/kubelet/pods/59936056-800e-4bb0-a80e-40d9194a4d5e/volumes"
Nov 03 01:24:14.026973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:14.026942    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:17.172737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:17.172687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:23.493185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:23.493141    4550 scope.go:122] "RemoveContainer" containerID="760d7b6bfb4b2415bd4bc2d42b64b77654fcf722aadd389c59a556a5950e3b9f"
Nov 03 01:24:24.727168 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:24:24.727124    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:24:29.614616 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:29.614572    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:24:36.038630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:36.038572    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:24:36.039328 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:24:36.038656    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:24:54.725475 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:24:54.725402    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:25:00.454774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:25:00.454742    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:25:01.853658 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:25:01.853622    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:25:17.273619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:25:17.273582    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:25:24.786727 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:25:24.786667    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:25:27.103884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:25:27.103842    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:25:54.721875 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:25:54.721789    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:26:17.410394 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:26:17.410352    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:26:24.551761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:26:24.551712    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:26:24.732203 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:26:24.732161    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:26:24.741962 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:26:24.741921    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:26:25.297513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:26:25.297474    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:26:54.721764 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:26:54.721724    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:27:17.497814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:27:17.497780    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:27:24.721891 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:27:24.721857    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:27:38.257655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:27:38.257614    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:27:38.340268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:27:38.340233    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:27:41.352256 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:27:41.352219    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:27:54.723232 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:27:54.723192    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:28:00.028134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:00.028096    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"]
Nov 03 01:28:00.041174 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:00.041141    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ea7e196-rmr8v"]
Nov 03 01:28:01.805688 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:01.805656    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24f226b-22f4-4304-81f8-9dab4fd1c9cc" path="/var/lib/kubelet/pods/c24f226b-22f4-4304-81f8-9dab4fd1c9cc/volumes"
Nov 03 01:28:02.519102 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:02.519065    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:28:02.617883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:02.617849    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:28:02.817685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:02.817595    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:28:10.887549 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:10.887518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:28:17.623111 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:17.623072    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:28:24.721139 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:28:24.721098    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:28:32.727154 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:28:32.727006    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:28:54.722707 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:28:54.722670    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:29:17.727738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:29:17.727700    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:29:23.558600 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:29:23.558569    4550 scope.go:122] "RemoveContainer" containerID="508909d14fdd80ce042c35fb6c7b0b935f772b4e6fd71a100b80d93e6bdf8c38"
Nov 03 01:29:24.724185 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:29:24.724133    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:29:54.723176 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:29:54.723132    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:30:04.734246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.734165    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"]
Nov 03 01:30:04.736630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.734592    4550 cpu_manager.go:414] "containerMap: removing container" podUID="59936056-800e-4bb0-a80e-40d9194a4d5e" containerName="revoke-keys"
Nov 03 01:30:04.736630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.734606    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59936056-800e-4bb0-a80e-40d9194a4d5e" containerName="revoke-keys"
Nov 03 01:30:04.736630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.734673    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="59936056-800e-4bb0-a80e-40d9194a4d5e" containerName="revoke-keys"
Nov 03 01:30:04.737582 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.737567    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.759142 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.759112    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"]
Nov 03 01:30:04.768054 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.768024    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:30:04.768230 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.768127    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:30:04.874143 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.874111    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.874342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.874231    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xpqj\" (UniqueName: \"kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.874342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.874271    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.974790 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.974746    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xpqj\" (UniqueName: \"kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.975006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.974801    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.975006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.974879    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.975566 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.975535    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:04.977539 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:04.977505    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:05.004229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:05.004156    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xpqj\" (UniqueName: \"kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj\") pod \"maas-api-revoke-sub-a4aef7d7-qg726\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:05.049611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:05.049569    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:05.342545 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:05.342513    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"]
Nov 03 01:30:05.343021 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:30:05.342996    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23b52ed_b9a7_4a46_b70f_a42c3186df60.slice/crio-4214fc06a8966d78d70c9a77ee19a6dbfc23c0f775169507a26aecbfea484c4d WatchSource:0}: Error finding container 4214fc06a8966d78d70c9a77ee19a6dbfc23c0f775169507a26aecbfea484c4d: Status 404 returned error can't find the container with id 4214fc06a8966d78d70c9a77ee19a6dbfc23c0f775169507a26aecbfea484c4d
Nov 03 01:30:06.148679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:06.148580    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726" event={"ID":"d23b52ed-b9a7-4a46-b70f-a42c3186df60","Type":"ContainerStarted","Data":"321ac9683e44d5c4daf06a5fec1bad694433641935fee31ad33d13cc6dfc06a7"}
Nov 03 01:30:06.149164 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:06.148679    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726" event={"ID":"d23b52ed-b9a7-4a46-b70f-a42c3186df60","Type":"ContainerStarted","Data":"4214fc06a8966d78d70c9a77ee19a6dbfc23c0f775169507a26aecbfea484c4d"}
Nov 03 01:30:06.189851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:06.189666    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726" podStartSLOduration=2.189649547 podStartE2EDuration="2.189649547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:06.187322861 +0000 UTC m=+5088.508859662" watchObservedRunningTime="2026-11-03 01:30:06.189649547 +0000 UTC m=+5088.511186343"
Nov 03 01:30:07.154238 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:07.154194    4550 generic.go:356] "Generic (PLEG): container finished" podID="d23b52ed-b9a7-4a46-b70f-a42c3186df60" containerID="321ac9683e44d5c4daf06a5fec1bad694433641935fee31ad33d13cc6dfc06a7" exitCode=0
Nov 03 01:30:07.154684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:07.154279    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726" event={"ID":"d23b52ed-b9a7-4a46-b70f-a42c3186df60","Type":"ContainerDied","Data":"321ac9683e44d5c4daf06a5fec1bad694433641935fee31ad33d13cc6dfc06a7"}
Nov 03 01:30:08.297504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.297476    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:08.406290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.406255    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca\") pod \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") "
Nov 03 01:30:08.406583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.406310    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca\") pod \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") "
Nov 03 01:30:08.406583 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.406405    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj\" (UniqueName: \"kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj\") pod \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\" (UID: \"d23b52ed-b9a7-4a46-b70f-a42c3186df60\") "
Nov 03 01:30:08.406774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.406749    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca" pod "d23b52ed-b9a7-4a46-b70f-a42c3186df60" (UID: "d23b52ed-b9a7-4a46-b70f-a42c3186df60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:30:08.406864 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.406840    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:30:08.408665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.408640    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca" pod "d23b52ed-b9a7-4a46-b70f-a42c3186df60" (UID: "d23b52ed-b9a7-4a46-b70f-a42c3186df60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:30:08.408776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.408696    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj" pod "d23b52ed-b9a7-4a46-b70f-a42c3186df60" (UID: "d23b52ed-b9a7-4a46-b70f-a42c3186df60"). InnerVolumeSpecName "kube-api-access-5xpqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:30:08.508186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.508150    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23b52ed-b9a7-4a46-b70f-a42c3186df60-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:30:08.508186 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:08.508179    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xpqj\" (UniqueName: \"kubernetes.io/projected/d23b52ed-b9a7-4a46-b70f-a42c3186df60-kube-api-access-5xpqj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:30:09.164570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:09.164537    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"
Nov 03 01:30:09.164738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:09.164565    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726" event={"ID":"d23b52ed-b9a7-4a46-b70f-a42c3186df60","Type":"ContainerDied","Data":"4214fc06a8966d78d70c9a77ee19a6dbfc23c0f775169507a26aecbfea484c4d"}
Nov 03 01:30:09.164738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:09.164614    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4214fc06a8966d78d70c9a77ee19a6dbfc23c0f775169507a26aecbfea484c4d"
Nov 03 01:30:17.855564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:17.855517    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:30:23.264094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:23.263545    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:30:23.264094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:23.263626    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:30:23.578176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:30:23.578087    4550 scope.go:122] "RemoveContainer" containerID="f45e326e4c653efa790fb328b593cd789bdff2cb214a302c2e911435e8b25656"
Nov 03 01:30:24.724271 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:30:24.724235    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:30:54.722871 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:30:54.722828    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:31:18.010227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:31:18.010189    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:31:24.722626 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:31:24.722588    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:31:54.723298 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:31:54.723218    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:32:15.095876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:15.095836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:15.555650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:15.555618    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:15.690858 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:15.690820    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:18.083755 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:18.083715    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:18.183988 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:18.183947    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:18.787312 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:18.787271    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:18.893711 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:18.893671    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:18.992127 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:18.992088    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:20.365620 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:20.365581    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:20.465300 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:20.465263    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:20.567368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:20.567325    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:20.679936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:20.679901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:22.121675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:22.121637    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:23.837336 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:23.837300    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:24.723314 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:32:24.723273    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:32:25.653541 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:25.653505    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:27.449078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:27.449044    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:29.257487 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:29.257454    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:31.030222 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:31.030186    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:32.824406 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:32.824369    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:33.703520 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:33.703483    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:34.286222 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:34.286180    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:34.395890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:34.395851    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:34.502743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:34.502705    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:34.604222 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:34.604139    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:35.485079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:35.485041    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:36.509491 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:36.509450    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:38.258112 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:38.258077    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:40.133515 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:40.133466    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:41.814444 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:41.814392    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:43.434022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:43.433989    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:45.141885 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:45.141852    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:46.836570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:46.836525    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:48.624366 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:48.624326    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:50.324388 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:50.324349    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:52.041701 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:52.041664    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:53.716025 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:53.715990    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:54.722834 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:32:54.722790    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:32:55.445267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:55.445231    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:57.114524 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:57.114488    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:32:58.739324 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:32:58.739292    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:00.447312 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:00.447223    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:02.227479 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:02.227447    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:03.912263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:03.912218    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:05.617254 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:05.617216    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:07.318208 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:07.318172    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:09.025116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:09.025083    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:10.728685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:10.728650    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:12.417828 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:12.417797    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:14.064998 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:14.064961    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:15.719446 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:15.719400    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:16.883601 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:16.883564    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:17.414221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:17.414192    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:17.518891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:17.518849    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:18.202378 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:18.202311    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:19.134577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:19.134518    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:20.933676 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:20.933640    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:21.945630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:21.945600    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:22.637914 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:22.637875    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:23.230037 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:23.229996    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:23.408245 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:23.408208    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:23.766440 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:23.766401    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:24.422736 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:24.422696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:24.724350 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:33:24.724259    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:33:25.842142 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:25.842107    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:29.279951 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:29.279920    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:29.373355 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:29.373323    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:54.722257 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:33:54.722220    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:33:56.503774 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:56.503739    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:56.608393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:56.608353    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:58.077987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:58.077948    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:58.246734 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:58.246694    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:58.359784 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:58.359696    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:58.711610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:58.711569    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:33:59.958421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:33:59.958383    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:00.079185 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:00.079147    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:01.819032 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:01.818991    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:03.421737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:03.421702    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:05.138947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:05.138910    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:05.932380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:05.932344    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:06.036897 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:06.036862    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:06.831048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:06.831000    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:08.623622 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:08.623592    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:10.241868 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:10.241836    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:11.926447 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:11.926397    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:13.621060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:13.621027    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:15.417732 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:15.417687    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:17.147289 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:17.147244    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:17.247788 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:17.247748    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:18.311146 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:18.311102    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:18.570630 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:18.570522    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:19.109750 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:19.109717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:19.210192 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:19.210153    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:19.655614 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:19.655577    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:34:24.724642 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:34:24.724606    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:34:39.058723 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:39.058687    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"]
Nov 03 01:34:39.059201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:39.058901    4550 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt" podUID="0b1877da-8248-4ab3-b721-c154ffa7e09f" containerName="manager" containerID="cri-o://cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17" gracePeriod=10
Nov 03 01:34:40.409963 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.409933    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 01:34:40.461276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.461190    4550 generic.go:356] "Generic (PLEG): container finished" podID="0b1877da-8248-4ab3-b721-c154ffa7e09f" containerID="cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17" exitCode=0
Nov 03 01:34:40.461276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.461235    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt" event={"ID":"0b1877da-8248-4ab3-b721-c154ffa7e09f","Type":"ContainerDied","Data":"cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17"}
Nov 03 01:34:40.461276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.461247    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"
Nov 03 01:34:40.461276 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.461279    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt" event={"ID":"0b1877da-8248-4ab3-b721-c154ffa7e09f","Type":"ContainerDied","Data":"961b106cbef5a60c03551c9752a6695c20fa6bebee5b94a43485f4b4880ecfbd"}
Nov 03 01:34:40.461592 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.461302    4550 scope.go:122] "RemoveContainer" containerID="cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17"
Nov 03 01:34:40.474051 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.474008    4550 scope.go:122] "RemoveContainer" containerID="cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17"
Nov 03 01:34:40.474351 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:34:40.474323    4550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17\": container with ID starting with cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17 not found: ID does not exist" containerID="cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17"
Nov 03 01:34:40.474418 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.474361    4550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17"} err="failed to get container status \"cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17\": rpc error: code = NotFound desc = could not find container \"cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17\": container with ID starting with cb61e85bbb65455c14a30a62e035957da2612f7f7557cd6f1c71a70082934a17 not found: ID does not exist"
Nov 03 01:34:40.495554 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.495513    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume\") pod \"0b1877da-8248-4ab3-b721-c154ffa7e09f\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") "
Nov 03 01:34:40.495766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.495668    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj\" (UniqueName: \"kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj\") pod \"0b1877da-8248-4ab3-b721-c154ffa7e09f\" (UID: \"0b1877da-8248-4ab3-b721-c154ffa7e09f\") "
Nov 03 01:34:40.495973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.495948    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume" pod "0b1877da-8248-4ab3-b721-c154ffa7e09f" (UID: "0b1877da-8248-4ab3-b721-c154ffa7e09f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:40.498162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.498129    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj" pod "0b1877da-8248-4ab3-b721-c154ffa7e09f" (UID: "0b1877da-8248-4ab3-b721-c154ffa7e09f"). InnerVolumeSpecName "kube-api-access-7lwxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:40.596850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.596811    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwxj\" (UniqueName: \"kubernetes.io/projected/0b1877da-8248-4ab3-b721-c154ffa7e09f-kube-api-access-7lwxj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:34:40.596850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.596844    4550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b1877da-8248-4ab3-b721-c154ffa7e09f-extensions-socket-volume\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:34:40.808619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.808584    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"]
Nov 03 01:34:40.821748 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:40.821718    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-89stt"]
Nov 03 01:34:41.805743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:34:41.805706    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1877da-8248-4ab3-b721-c154ffa7e09f" path="/var/lib/kubelet/pods/0b1877da-8248-4ab3-b721-c154ffa7e09f/volumes"
Nov 03 01:34:54.723204 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:34:54.723164    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:35:09.035576 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:09.035534    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"]
Nov 03 01:35:09.047930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:09.047891    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-qg726"]
Nov 03 01:35:09.057699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:09.057663    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:35:09.812144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:09.812108    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23b52ed-b9a7-4a46-b70f-a42c3186df60" path="/var/lib/kubelet/pods/d23b52ed-b9a7-4a46-b70f-a42c3186df60/volumes"
Nov 03 01:35:18.426909 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:18.426871    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:35:19.693570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:19.693536    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:35:20.475122 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:20.475078    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:35:24.723600 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:35:24.723556    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:35:42.126446 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.126396    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:35:42.228746 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.228670    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:35:42.240623 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.240594    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"]
Nov 03 01:35:42.240947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.240934    4550 cpu_manager.go:414] "containerMap: removing container" podUID="d23b52ed-b9a7-4a46-b70f-a42c3186df60" containerName="revoke-keys"
Nov 03 01:35:42.240993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.240949    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23b52ed-b9a7-4a46-b70f-a42c3186df60" containerName="revoke-keys"
Nov 03 01:35:42.240993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.240957    4550 cpu_manager.go:414] "containerMap: removing container" podUID="0b1877da-8248-4ab3-b721-c154ffa7e09f" containerName="manager"
Nov 03 01:35:42.240993 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.240963    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1877da-8248-4ab3-b721-c154ffa7e09f" containerName="manager"
Nov 03 01:35:42.241093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.241037    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23b52ed-b9a7-4a46-b70f-a42c3186df60" containerName="revoke-keys"
Nov 03 01:35:42.241093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.241047    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1877da-8248-4ab3-b721-c154ffa7e09f" containerName="manager"
Nov 03 01:35:42.244147 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.244125    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.259761 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.259728    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"]
Nov 03 01:35:42.271131 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.271094    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l2kb6\""
Nov 03 01:35:42.375721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.375686    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b3a7806-8c9f-4e02-9848-cc488d0622ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f4r7b\" (UID: \"3b3a7806-8c9f-4e02-9848-cc488d0622ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.375928 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.375743    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnr6z\" (UniqueName: \"kubernetes.io/projected/3b3a7806-8c9f-4e02-9848-cc488d0622ba-kube-api-access-lnr6z\") pod \"kuadrant-operator-controller-manager-678d87766f-f4r7b\" (UID: \"3b3a7806-8c9f-4e02-9848-cc488d0622ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.476271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.476223    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b3a7806-8c9f-4e02-9848-cc488d0622ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f4r7b\" (UID: \"3b3a7806-8c9f-4e02-9848-cc488d0622ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.476513 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.476298    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnr6z\" (UniqueName: \"kubernetes.io/projected/3b3a7806-8c9f-4e02-9848-cc488d0622ba-kube-api-access-lnr6z\") pod \"kuadrant-operator-controller-manager-678d87766f-f4r7b\" (UID: \"3b3a7806-8c9f-4e02-9848-cc488d0622ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.476668 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.476646    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b3a7806-8c9f-4e02-9848-cc488d0622ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-f4r7b\" (UID: \"3b3a7806-8c9f-4e02-9848-cc488d0622ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.500330 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.500295    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnr6z\" (UniqueName: \"kubernetes.io/projected/3b3a7806-8c9f-4e02-9848-cc488d0622ba-kube-api-access-lnr6z\") pod \"kuadrant-operator-controller-manager-678d87766f-f4r7b\" (UID: \"3b3a7806-8c9f-4e02-9848-cc488d0622ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.556945 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.556910    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:42.879523 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:42.879495    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"]
Nov 03 01:35:42.879945 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:35:42.879918    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3a7806_8c9f_4e02_9848_cc488d0622ba.slice/crio-03c408b0241becd7039d6039fec404bc062f1e0d08941e9415d73dfa51cdc244 WatchSource:0}: Error finding container 03c408b0241becd7039d6039fec404bc062f1e0d08941e9415d73dfa51cdc244: Status 404 returned error can't find the container with id 03c408b0241becd7039d6039fec404bc062f1e0d08941e9415d73dfa51cdc244
Nov 03 01:35:43.736411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:43.736370    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b" event={"ID":"3b3a7806-8c9f-4e02-9848-cc488d0622ba","Type":"ContainerStarted","Data":"15a63e2bca6bea3d59ecd593978bb5048dc8100f10a79a41203d8f614aad7f75"}
Nov 03 01:35:43.736834 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:43.736602    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:35:43.736834 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:43.736645    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b" event={"ID":"3b3a7806-8c9f-4e02-9848-cc488d0622ba","Type":"ContainerStarted","Data":"03c408b0241becd7039d6039fec404bc062f1e0d08941e9415d73dfa51cdc244"}
Nov 03 01:35:43.808322 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:43.808258    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b" podStartSLOduration=1.808242979 podStartE2EDuration="1.808242979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:43.793952155 +0000 UTC m=+5426.115488956" watchObservedRunningTime="2026-11-03 01:35:43.808242979 +0000 UTC m=+5426.129779775"
Nov 03 01:35:54.723426 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:35:54.723383    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:35:54.742848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:35:54.742811    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-f4r7b"
Nov 03 01:36:18.558225 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:18.558180    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:36:23.226543 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:23.226493    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:23.227211 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:23.227173    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:23.643655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:23.643555    4550 scope.go:122] "RemoveContainer" containerID="321ac9683e44d5c4daf06a5fec1bad694433641935fee31ad33d13cc6dfc06a7"
Nov 03 01:36:24.723704 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:36:24.723665    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:36:37.115806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:37.115759    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:36:51.036756 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:51.036694    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:36:51.037506 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:51.037473    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:36:53.454560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.454520    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"]
Nov 03 01:36:53.458311 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.458284    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.476486 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.476446    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"]
Nov 03 01:36:53.487534 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.487500    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:36:53.487724 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.487552    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:36:53.596500 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.596446    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ssg\" (UniqueName: \"kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.596743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.596589    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.596743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.596681    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.698022 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.697978    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.698267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.698041    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ssg\" (UniqueName: \"kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.698267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.698088    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.698733 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.698708    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.700538 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.700513    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.718526 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.718457    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ssg\" (UniqueName: \"kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg\") pod \"maas-api-revoke-sub-218ade40-wzn6p\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:53.771953 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:53.771911    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:54.067605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:54.067570    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"]
Nov 03 01:36:54.068035 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:36:54.068013    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b8d459_dd3f_4cb9_812b_82312bfe7607.slice/crio-548a5cd9f54781c4142a05a8012ad8b90bad8830aab8f0f1df8305809468cca9 WatchSource:0}: Error finding container 548a5cd9f54781c4142a05a8012ad8b90bad8830aab8f0f1df8305809468cca9: Status 404 returned error can't find the container with id 548a5cd9f54781c4142a05a8012ad8b90bad8830aab8f0f1df8305809468cca9
Nov 03 01:36:54.724109 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:36:54.724071    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:36:55.073381 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:55.073289    4550 generic.go:356] "Generic (PLEG): container finished" podID="77b8d459-dd3f-4cb9-812b-82312bfe7607" containerID="7ffddb85f9bc284ea6d2eb61928fb536b371a732e88e85d4362e0ea4a7c8fa4d" exitCode=0
Nov 03 01:36:55.073589 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:55.073378    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p" event={"ID":"77b8d459-dd3f-4cb9-812b-82312bfe7607","Type":"ContainerDied","Data":"7ffddb85f9bc284ea6d2eb61928fb536b371a732e88e85d4362e0ea4a7c8fa4d"}
Nov 03 01:36:55.073589 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:55.073453    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p" event={"ID":"77b8d459-dd3f-4cb9-812b-82312bfe7607","Type":"ContainerStarted","Data":"548a5cd9f54781c4142a05a8012ad8b90bad8830aab8f0f1df8305809468cca9"}
Nov 03 01:36:56.221777 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.221749    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:56.320359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.320322    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca\") pod \"77b8d459-dd3f-4cb9-812b-82312bfe7607\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") "
Nov 03 01:36:56.320561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.320388    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca\") pod \"77b8d459-dd3f-4cb9-812b-82312bfe7607\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") "
Nov 03 01:36:56.320561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.320513    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg\" (UniqueName: \"kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg\") pod \"77b8d459-dd3f-4cb9-812b-82312bfe7607\" (UID: \"77b8d459-dd3f-4cb9-812b-82312bfe7607\") "
Nov 03 01:36:56.320840 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.320804    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca" pod "77b8d459-dd3f-4cb9-812b-82312bfe7607" (UID: "77b8d459-dd3f-4cb9-812b-82312bfe7607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:56.322512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.322485    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca" pod "77b8d459-dd3f-4cb9-812b-82312bfe7607" (UID: "77b8d459-dd3f-4cb9-812b-82312bfe7607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:56.322628 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.322517    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg" pod "77b8d459-dd3f-4cb9-812b-82312bfe7607" (UID: "77b8d459-dd3f-4cb9-812b-82312bfe7607"). InnerVolumeSpecName "kube-api-access-k7ssg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:56.421936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.421896    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7ssg\" (UniqueName: \"kubernetes.io/projected/77b8d459-dd3f-4cb9-812b-82312bfe7607-kube-api-access-k7ssg\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:36:56.421936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.421929    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:36:56.421936 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:56.421938    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b8d459-dd3f-4cb9-812b-82312bfe7607-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:36:57.083530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:57.083490    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p" event={"ID":"77b8d459-dd3f-4cb9-812b-82312bfe7607","Type":"ContainerDied","Data":"548a5cd9f54781c4142a05a8012ad8b90bad8830aab8f0f1df8305809468cca9"}
Nov 03 01:36:57.083530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:57.083527    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"
Nov 03 01:36:57.083768 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:36:57.083554    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="548a5cd9f54781c4142a05a8012ad8b90bad8830aab8f0f1df8305809468cca9"
Nov 03 01:37:03.300349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.300299    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"]
Nov 03 01:37:03.301023 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.300996    4550 cpu_manager.go:414] "containerMap: removing container" podUID="77b8d459-dd3f-4cb9-812b-82312bfe7607" containerName="revoke-keys"
Nov 03 01:37:03.301093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.301028    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b8d459-dd3f-4cb9-812b-82312bfe7607" containerName="revoke-keys"
Nov 03 01:37:03.301182 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.301168    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b8d459-dd3f-4cb9-812b-82312bfe7607" containerName="revoke-keys"
Nov 03 01:37:03.305313 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.305281    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.322199 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.322160    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:37:03.322627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.322602    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:37:03.324699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.324634    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"]
Nov 03 01:37:03.391847 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.391802    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.392051 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.391857    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.392051 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.392012    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxd6\" (UniqueName: \"kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.493281 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.493238    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.493485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.493322    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxd6\" (UniqueName: \"kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.493485 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.493400    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.493992 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.493964    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.495844 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.495812    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.512971 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.512937    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxd6\" (UniqueName: \"kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6\") pod \"maas-api-revoke-sub-8c1aaced-8nfnt\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.627180 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.627085    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:03.921260 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:03.921233    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"]
Nov 03 01:37:03.921880 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:37:03.921854    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b711c1_11cd_467c_b474_273f22eb0cad.slice/crio-d6bc25d49a3ed16e0b93f6481c63f65f4faa3f0db8a7a2803f828a806d2be4ae WatchSource:0}: Error finding container d6bc25d49a3ed16e0b93f6481c63f65f4faa3f0db8a7a2803f828a806d2be4ae: Status 404 returned error can't find the container with id d6bc25d49a3ed16e0b93f6481c63f65f4faa3f0db8a7a2803f828a806d2be4ae
Nov 03 01:37:04.114385 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:04.114349    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt" event={"ID":"a1b711c1-11cd-467c-b474-273f22eb0cad","Type":"ContainerStarted","Data":"d6bc25d49a3ed16e0b93f6481c63f65f4faa3f0db8a7a2803f828a806d2be4ae"}
Nov 03 01:37:05.119662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:05.119573    4550 generic.go:356] "Generic (PLEG): container finished" podID="a1b711c1-11cd-467c-b474-273f22eb0cad" containerID="8ccc34414553c1cd5b3407914065d3f9925c9e2bf2a1b590661c42c7b5541951" exitCode=0
Nov 03 01:37:05.119662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:05.119644    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt" event={"ID":"a1b711c1-11cd-467c-b474-273f22eb0cad","Type":"ContainerDied","Data":"8ccc34414553c1cd5b3407914065d3f9925c9e2bf2a1b590661c42c7b5541951"}
Nov 03 01:37:06.283211 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.283177    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:06.423740 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.423708    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6\" (UniqueName: \"kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6\") pod \"a1b711c1-11cd-467c-b474-273f22eb0cad\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") "
Nov 03 01:37:06.423924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.423775    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca\") pod \"a1b711c1-11cd-467c-b474-273f22eb0cad\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") "
Nov 03 01:37:06.423924 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.423798    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca\") pod \"a1b711c1-11cd-467c-b474-273f22eb0cad\" (UID: \"a1b711c1-11cd-467c-b474-273f22eb0cad\") "
Nov 03 01:37:06.424266 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.424243    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca" pod "a1b711c1-11cd-467c-b474-273f22eb0cad" (UID: "a1b711c1-11cd-467c-b474-273f22eb0cad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:37:06.425906 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.425884    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca" pod "a1b711c1-11cd-467c-b474-273f22eb0cad" (UID: "a1b711c1-11cd-467c-b474-273f22eb0cad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:37:06.426007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.425984    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6" pod "a1b711c1-11cd-467c-b474-273f22eb0cad" (UID: "a1b711c1-11cd-467c-b474-273f22eb0cad"). InnerVolumeSpecName "kube-api-access-lrxd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:06.525169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.525122    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrxd6\" (UniqueName: \"kubernetes.io/projected/a1b711c1-11cd-467c-b474-273f22eb0cad-kube-api-access-lrxd6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:37:06.525169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.525162    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:37:06.525169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:06.525178    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1b711c1-11cd-467c-b474-273f22eb0cad-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:37:07.131390 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:07.131354    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt" event={"ID":"a1b711c1-11cd-467c-b474-273f22eb0cad","Type":"ContainerDied","Data":"d6bc25d49a3ed16e0b93f6481c63f65f4faa3f0db8a7a2803f828a806d2be4ae"}
Nov 03 01:37:07.131597 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:07.131408    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6bc25d49a3ed16e0b93f6481c63f65f4faa3f0db8a7a2803f828a806d2be4ae"
Nov 03 01:37:07.131597 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:07.131370    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"
Nov 03 01:37:18.659303 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:18.659256    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:37:24.723541 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:37:24.723495    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:37:36.304806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.304702    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"]
Nov 03 01:37:36.305296 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.305192    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a1b711c1-11cd-467c-b474-273f22eb0cad" containerName="revoke-keys"
Nov 03 01:37:36.305296 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.305213    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b711c1-11cd-467c-b474-273f22eb0cad" containerName="revoke-keys"
Nov 03 01:37:36.305457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.305317    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b711c1-11cd-467c-b474-273f22eb0cad" containerName="revoke-keys"
Nov 03 01:37:36.308523 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.308500    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.326901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.326866    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"]
Nov 03 01:37:36.337090 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.337056    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:37:36.337278 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.337112    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:37:36.455119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.455083    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlhj\" (UniqueName: \"kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.455308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.455137    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.455308 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.455250    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.556598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.556515    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.556598 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.556578    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.556794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.556639    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlhj\" (UniqueName: \"kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.557197 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.557172    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.559005 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.558980    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.577035 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.577001    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlhj\" (UniqueName: \"kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj\") pod \"maas-api-revoke-sub-e28ec5c3-dlc6p\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.621849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.621813    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:36.919667 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:36.919634    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"]
Nov 03 01:37:36.921096 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:37:36.921064    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcda6729_715b_460f_889c_1993d9ab6570.slice/crio-4fa0fce791234022809af7a1c1a890449932c515e3e2fab9790b45f409218b73 WatchSource:0}: Error finding container 4fa0fce791234022809af7a1c1a890449932c515e3e2fab9790b45f409218b73: Status 404 returned error can't find the container with id 4fa0fce791234022809af7a1c1a890449932c515e3e2fab9790b45f409218b73
Nov 03 01:37:37.273126 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:37.273030    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p" event={"ID":"bcda6729-715b-460f-889c-1993d9ab6570","Type":"ContainerStarted","Data":"e6aee83f75720d227ebe73e6486c17d9b23e952ed3c9976f39366e812f7a1ca4"}
Nov 03 01:37:37.273126 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:37.273092    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p" event={"ID":"bcda6729-715b-460f-889c-1993d9ab6570","Type":"ContainerStarted","Data":"4fa0fce791234022809af7a1c1a890449932c515e3e2fab9790b45f409218b73"}
Nov 03 01:37:37.313017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:37.312957    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p" podStartSLOduration=1.312936441 podStartE2EDuration="1.312936441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:37:37.312617158 +0000 UTC m=+5539.634153957" watchObservedRunningTime="2026-11-03 01:37:37.312936441 +0000 UTC m=+5539.634473241"
Nov 03 01:37:38.278273 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:38.278235    4550 generic.go:356] "Generic (PLEG): container finished" podID="bcda6729-715b-460f-889c-1993d9ab6570" containerID="e6aee83f75720d227ebe73e6486c17d9b23e952ed3c9976f39366e812f7a1ca4" exitCode=0
Nov 03 01:37:38.278490 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:38.278318    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p" event={"ID":"bcda6729-715b-460f-889c-1993d9ab6570","Type":"ContainerDied","Data":"e6aee83f75720d227ebe73e6486c17d9b23e952ed3c9976f39366e812f7a1ca4"}
Nov 03 01:37:38.798766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:38.798717    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:37:39.429626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.429596    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:39.586361 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.586250    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca\") pod \"bcda6729-715b-460f-889c-1993d9ab6570\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") "
Nov 03 01:37:39.586361 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.586306    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj\" (UniqueName: \"kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj\") pod \"bcda6729-715b-460f-889c-1993d9ab6570\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") "
Nov 03 01:37:39.586657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.586521    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca\") pod \"bcda6729-715b-460f-889c-1993d9ab6570\" (UID: \"bcda6729-715b-460f-889c-1993d9ab6570\") "
Nov 03 01:37:39.586867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.586841    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca" pod "bcda6729-715b-460f-889c-1993d9ab6570" (UID: "bcda6729-715b-460f-889c-1993d9ab6570"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:37:39.588380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.588353    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca" pod "bcda6729-715b-460f-889c-1993d9ab6570" (UID: "bcda6729-715b-460f-889c-1993d9ab6570"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:37:39.588517 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.588461    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj" pod "bcda6729-715b-460f-889c-1993d9ab6570" (UID: "bcda6729-715b-460f-889c-1993d9ab6570"). InnerVolumeSpecName "kube-api-access-kdlhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:37:39.688119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.688073    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:37:39.688119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.688112    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcda6729-715b-460f-889c-1993d9ab6570-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:37:39.688119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:39.688124    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlhj\" (UniqueName: \"kubernetes.io/projected/bcda6729-715b-460f-889c-1993d9ab6570-kube-api-access-kdlhj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:37:40.288331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:40.288291    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"
Nov 03 01:37:40.288331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:40.288320    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p" event={"ID":"bcda6729-715b-460f-889c-1993d9ab6570","Type":"ContainerDied","Data":"4fa0fce791234022809af7a1c1a890449932c515e3e2fab9790b45f409218b73"}
Nov 03 01:37:40.288878 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:37:40.288375    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa0fce791234022809af7a1c1a890449932c515e3e2fab9790b45f409218b73"
Nov 03 01:37:54.823269 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:37:54.823224    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:38:07.066789 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.066742    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"]
Nov 03 01:38:07.068229 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.068197    4550 cpu_manager.go:414] "containerMap: removing container" podUID="bcda6729-715b-460f-889c-1993d9ab6570" containerName="revoke-keys"
Nov 03 01:38:07.068421 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.068393    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcda6729-715b-460f-889c-1993d9ab6570" containerName="revoke-keys"
Nov 03 01:38:07.068738 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.068722    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcda6729-715b-460f-889c-1993d9ab6570" containerName="revoke-keys"
Nov 03 01:38:07.073640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.073608    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.086304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.086267    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"]
Nov 03 01:38:07.096077 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.096039    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:07.096290 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.096078    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:38:07.104725 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.104683    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.104925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.104818    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.104925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.104886    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgntv\" (UniqueName: \"kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.205743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.205704    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.205930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.205758    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgntv\" (UniqueName: \"kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.205930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.205828    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.206577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.206550    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.208193 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.208169    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.226492 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.226456    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgntv\" (UniqueName: \"kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv\") pod \"maas-api-revoke-sub-aeb7e7e4-6clw6\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.390349 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.390256    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:07.684959 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:07.684913    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"]
Nov 03 01:38:07.685153 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:38:07.685130    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815949c5_ac36_4022_aa58_1fbd37b6df9c.slice/crio-da055fb14f9a7758382043645e5cddce874cb784928a6c9b3cf7f7e625c9edd5 WatchSource:0}: Error finding container da055fb14f9a7758382043645e5cddce874cb784928a6c9b3cf7f7e625c9edd5: Status 404 returned error can't find the container with id da055fb14f9a7758382043645e5cddce874cb784928a6c9b3cf7f7e625c9edd5
Nov 03 01:38:08.433688 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:08.433646    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6" event={"ID":"815949c5-ac36-4022-aa58-1fbd37b6df9c","Type":"ContainerStarted","Data":"4ddb8af49b823d527b0377f84322397d13c4bd14b4c866ba55b84b17dace1101"}
Nov 03 01:38:08.434096 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:08.433707    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6" event={"ID":"815949c5-ac36-4022-aa58-1fbd37b6df9c","Type":"ContainerStarted","Data":"da055fb14f9a7758382043645e5cddce874cb784928a6c9b3cf7f7e625c9edd5"}
Nov 03 01:38:08.475852 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:08.475784    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6" podStartSLOduration=1.475768848 podStartE2EDuration="1.475768848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:08.474093366 +0000 UTC m=+5570.795630229" watchObservedRunningTime="2026-11-03 01:38:08.475768848 +0000 UTC m=+5570.797305646"
Nov 03 01:38:09.440144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:09.440107    4550 generic.go:356] "Generic (PLEG): container finished" podID="815949c5-ac36-4022-aa58-1fbd37b6df9c" containerID="4ddb8af49b823d527b0377f84322397d13c4bd14b4c866ba55b84b17dace1101" exitCode=0
Nov 03 01:38:09.440564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:09.440178    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6" event={"ID":"815949c5-ac36-4022-aa58-1fbd37b6df9c","Type":"ContainerDied","Data":"4ddb8af49b823d527b0377f84322397d13c4bd14b4c866ba55b84b17dace1101"}
Nov 03 01:38:10.570954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.570924    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:10.637831 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.637798    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca\") pod \"815949c5-ac36-4022-aa58-1fbd37b6df9c\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") "
Nov 03 01:38:10.638015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.637866    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv\" (UniqueName: \"kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv\") pod \"815949c5-ac36-4022-aa58-1fbd37b6df9c\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") "
Nov 03 01:38:10.638015 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.637956    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca\") pod \"815949c5-ac36-4022-aa58-1fbd37b6df9c\" (UID: \"815949c5-ac36-4022-aa58-1fbd37b6df9c\") "
Nov 03 01:38:10.638307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.638276    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca" pod "815949c5-ac36-4022-aa58-1fbd37b6df9c" (UID: "815949c5-ac36-4022-aa58-1fbd37b6df9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:10.640134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.640108    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca" pod "815949c5-ac36-4022-aa58-1fbd37b6df9c" (UID: "815949c5-ac36-4022-aa58-1fbd37b6df9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:10.640217 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.640148    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv" pod "815949c5-ac36-4022-aa58-1fbd37b6df9c" (UID: "815949c5-ac36-4022-aa58-1fbd37b6df9c"). InnerVolumeSpecName "kube-api-access-xgntv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:10.738685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.738598    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:10.738685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.738632    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/815949c5-ac36-4022-aa58-1fbd37b6df9c-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:10.738685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:10.738643    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgntv\" (UniqueName: \"kubernetes.io/projected/815949c5-ac36-4022-aa58-1fbd37b6df9c-kube-api-access-xgntv\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:11.450762 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:11.450729    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6"
Nov 03 01:38:11.450997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:11.450767    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeb7e7e4-6clw6" event={"ID":"815949c5-ac36-4022-aa58-1fbd37b6df9c","Type":"ContainerDied","Data":"da055fb14f9a7758382043645e5cddce874cb784928a6c9b3cf7f7e625c9edd5"}
Nov 03 01:38:11.450997 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:11.450821    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da055fb14f9a7758382043645e5cddce874cb784928a6c9b3cf7f7e625c9edd5"
Nov 03 01:38:17.428998 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.428957    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"]
Nov 03 01:38:17.429655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.429452    4550 cpu_manager.go:414] "containerMap: removing container" podUID="815949c5-ac36-4022-aa58-1fbd37b6df9c" containerName="revoke-keys"
Nov 03 01:38:17.429655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.429473    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815949c5-ac36-4022-aa58-1fbd37b6df9c" containerName="revoke-keys"
Nov 03 01:38:17.429655 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.429611    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="815949c5-ac36-4022-aa58-1fbd37b6df9c" containerName="revoke-keys"
Nov 03 01:38:17.432902 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.432879    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.452794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.452751    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"]
Nov 03 01:38:17.462699 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.462666    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:17.462929 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.462722    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:38:17.503590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.503519    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbczn\" (UniqueName: \"kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.503590 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.503589    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.503816 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.503730    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.605116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.605069    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbczn\" (UniqueName: \"kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.605116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.605118    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.605365 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.605314    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.605985 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.605959    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.607913 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.607884    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.625259 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.625222    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbczn\" (UniqueName: \"kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn\") pod \"maas-api-revoke-sub-c2fd3996-2jwgf\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:17.747163 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:17.747072    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:18.045369 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:18.045330    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"]
Nov 03 01:38:18.046118 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:38:18.046085    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd904cf_530a_4530_ab31_d690451bbcdc.slice/crio-506a765b2da51813cdd8abe87be37ee134cef53a83c4ffe2824e4a4716027167 WatchSource:0}: Error finding container 506a765b2da51813cdd8abe87be37ee134cef53a83c4ffe2824e4a4716027167: Status 404 returned error can't find the container with id 506a765b2da51813cdd8abe87be37ee134cef53a83c4ffe2824e4a4716027167
Nov 03 01:38:18.486684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:18.486644    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf" event={"ID":"fbd904cf-530a-4530-ab31-d690451bbcdc","Type":"ContainerStarted","Data":"4639ea99d5567327f2e00a04119c07097aa2ae801119cb6c73b56b8d4a8c2fce"}
Nov 03 01:38:18.487106 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:18.486735    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf" event={"ID":"fbd904cf-530a-4530-ab31-d690451bbcdc","Type":"ContainerStarted","Data":"506a765b2da51813cdd8abe87be37ee134cef53a83c4ffe2824e4a4716027167"}
Nov 03 01:38:18.525709 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:18.525650    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf" podStartSLOduration=1.525636189 podStartE2EDuration="1.525636189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:38:18.524894271 +0000 UTC m=+5580.846431095" watchObservedRunningTime="2026-11-03 01:38:18.525636189 +0000 UTC m=+5580.847172988"
Nov 03 01:38:18.867528 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:18.867409    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:38:19.493662 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:19.493629    4550 generic.go:356] "Generic (PLEG): container finished" podID="fbd904cf-530a-4530-ab31-d690451bbcdc" containerID="4639ea99d5567327f2e00a04119c07097aa2ae801119cb6c73b56b8d4a8c2fce" exitCode=0
Nov 03 01:38:19.494144 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:19.493711    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf" event={"ID":"fbd904cf-530a-4530-ab31-d690451bbcdc","Type":"ContainerDied","Data":"4639ea99d5567327f2e00a04119c07097aa2ae801119cb6c73b56b8d4a8c2fce"}
Nov 03 01:38:20.638649 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.638621    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:20.735267 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.735228    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca\") pod \"fbd904cf-530a-4530-ab31-d690451bbcdc\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") "
Nov 03 01:38:20.735508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.735311    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn\" (UniqueName: \"kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn\") pod \"fbd904cf-530a-4530-ab31-d690451bbcdc\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") "
Nov 03 01:38:20.735508 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.735348    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca\") pod \"fbd904cf-530a-4530-ab31-d690451bbcdc\" (UID: \"fbd904cf-530a-4530-ab31-d690451bbcdc\") "
Nov 03 01:38:20.735798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.735771    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca" pod "fbd904cf-530a-4530-ab31-d690451bbcdc" (UID: "fbd904cf-530a-4530-ab31-d690451bbcdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:20.737537 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.737515    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn" pod "fbd904cf-530a-4530-ab31-d690451bbcdc" (UID: "fbd904cf-530a-4530-ab31-d690451bbcdc"). InnerVolumeSpecName "kube-api-access-bbczn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:20.737640 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.737624    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca" pod "fbd904cf-530a-4530-ab31-d690451bbcdc" (UID: "fbd904cf-530a-4530-ab31-d690451bbcdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:20.836709 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.836617    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:20.836709 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.836653    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd904cf-530a-4530-ab31-d690451bbcdc-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:20.836709 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:20.836664    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbczn\" (UniqueName: \"kubernetes.io/projected/fbd904cf-530a-4530-ab31-d690451bbcdc-kube-api-access-bbczn\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:21.503744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:21.503702    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf" event={"ID":"fbd904cf-530a-4530-ab31-d690451bbcdc","Type":"ContainerDied","Data":"506a765b2da51813cdd8abe87be37ee134cef53a83c4ffe2824e4a4716027167"}
Nov 03 01:38:21.503744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:21.503743    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2fd3996-2jwgf"
Nov 03 01:38:21.503970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:21.503763    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="506a765b2da51813cdd8abe87be37ee134cef53a83c4ffe2824e4a4716027167"
Nov 03 01:38:23.776134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:23.776090    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:38:24.723816 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:38:24.723777    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:38:31.557888 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:31.557843    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:38:36.040685 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:36.040639    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-44ncj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:38:36.041357 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:36.041304    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-44ncj" podUID="f174f791-f438-4076-a30c-ecaca62452e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:38:42.276058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:42.276018    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:38:48.864403 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.864367    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"]
Nov 03 01:38:48.864819 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.864792    4550 cpu_manager.go:414] "containerMap: removing container" podUID="fbd904cf-530a-4530-ab31-d690451bbcdc" containerName="revoke-keys"
Nov 03 01:38:48.864819 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.864808    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd904cf-530a-4530-ab31-d690451bbcdc" containerName="revoke-keys"
Nov 03 01:38:48.864893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.864888    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd904cf-530a-4530-ab31-d690451bbcdc" containerName="revoke-keys"
Nov 03 01:38:48.868120 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.868089    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.879942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.879901    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.880134 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.880113    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.880183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.880144    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhsp9\" (UniqueName: \"kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.885200 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.885155    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:48.898466 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.898399    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:38:48.900249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.900221    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"]
Nov 03 01:38:48.980876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.980834    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.980876 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.980878    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhsp9\" (UniqueName: \"kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.981194 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.980912    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.981638 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.981614    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:48.983373 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:48.983344    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:49.001218 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:49.001189    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhsp9\" (UniqueName: \"kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9\") pod \"maas-api-revoke-sub-3748db8a-kfnpv\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:49.181509 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:49.181464    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:49.478581 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:38:49.478552    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937ffed2_8cc8_4eb1_a75a_0f7f1e8da76f.slice/crio-ea54fdaa952564734a4858d08909f63efb7919dc181ab79de5970c4d45d5b44d WatchSource:0}: Error finding container ea54fdaa952564734a4858d08909f63efb7919dc181ab79de5970c4d45d5b44d: Status 404 returned error can't find the container with id ea54fdaa952564734a4858d08909f63efb7919dc181ab79de5970c4d45d5b44d
Nov 03 01:38:49.479060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:49.479037    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"]
Nov 03 01:38:49.675109 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:49.675068    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv" event={"ID":"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f","Type":"ContainerStarted","Data":"ea54fdaa952564734a4858d08909f63efb7919dc181ab79de5970c4d45d5b44d"}
Nov 03 01:38:50.680814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:50.680776    4550 generic.go:356] "Generic (PLEG): container finished" podID="937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" containerID="d4d2ccc6ac9e1a6cd325bb9652d8fcb848e8b5a7f4a0faa648f5807a3127c07c" exitCode=0
Nov 03 01:38:50.681226 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:50.680833    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv" event={"ID":"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f","Type":"ContainerDied","Data":"d4d2ccc6ac9e1a6cd325bb9652d8fcb848e8b5a7f4a0faa648f5807a3127c07c"}
Nov 03 01:38:51.817084 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.817059    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:51.907002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.906960    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9\" (UniqueName: \"kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9\") pod \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") "
Nov 03 01:38:51.907002 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.907018    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca\") pod \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") "
Nov 03 01:38:51.907244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.907071    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca\") pod \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\" (UID: \"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f\") "
Nov 03 01:38:51.907531 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.907503    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca" pod "937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" (UID: "937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:38:51.909163 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.909138    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca" pod "937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" (UID: "937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:38:51.909277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:51.909205    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9" pod "937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" (UID: "937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f"). InnerVolumeSpecName "kube-api-access-mhsp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:38:52.008145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:52.008056    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:52.008145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:52.008092    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:52.008145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:52.008101    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhsp9\" (UniqueName: \"kubernetes.io/projected/937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f-kube-api-access-mhsp9\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:38:52.690961 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:52.690929    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv"
Nov 03 01:38:52.691162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:52.690925    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3748db8a-kfnpv" event={"ID":"937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f","Type":"ContainerDied","Data":"ea54fdaa952564734a4858d08909f63efb7919dc181ab79de5970c4d45d5b44d"}
Nov 03 01:38:52.691162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:52.691055    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea54fdaa952564734a4858d08909f63efb7919dc181ab79de5970c4d45d5b44d"
Nov 03 01:38:54.723516 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:38:54.723475    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:38:59.339767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.339721    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"]
Nov 03 01:38:59.340444 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.340404    4550 cpu_manager.go:414] "containerMap: removing container" podUID="937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" containerName="revoke-keys"
Nov 03 01:38:59.340550 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.340446    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" containerName="revoke-keys"
Nov 03 01:38:59.340611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.340594    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f" containerName="revoke-keys"
Nov 03 01:38:59.344060 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.344032    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.360294 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.360256    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"]
Nov 03 01:38:59.370687 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.370643    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:38:59.370896 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.370782    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:38:59.378331 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.378275    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.378677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.378342    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.378677 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.378571    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbnnj\" (UniqueName: \"kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.480082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.480008    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.480082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.480041    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.480082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.480076    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbnnj\" (UniqueName: \"kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.480827 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.480801    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.482611 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.482586    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.500346 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.500310    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbnnj\" (UniqueName: \"kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj\") pod \"maas-api-revoke-sub-f0877206-zsfjb\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.659981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.659942    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:38:59.957563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:38:59.957520    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"]
Nov 03 01:38:59.958062 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:38:59.958028    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978a4fec_a879_49ed_81e7_b1127cebbde6.slice/crio-91c7ba6626db87ef1f11154aae2521afdf3885c0072afebb22e5094ee79466ac WatchSource:0}: Error finding container 91c7ba6626db87ef1f11154aae2521afdf3885c0072afebb22e5094ee79466ac: Status 404 returned error can't find the container with id 91c7ba6626db87ef1f11154aae2521afdf3885c0072afebb22e5094ee79466ac
Nov 03 01:39:00.737376 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:00.737319    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb" event={"ID":"978a4fec-a879-49ed-81e7-b1127cebbde6","Type":"ContainerStarted","Data":"9b7ba6b51f98de92dec7b18c615ff24b9d6a47882a26a38d516d3fb198bc24ac"}
Nov 03 01:39:00.737930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:00.737393    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb" event={"ID":"978a4fec-a879-49ed-81e7-b1127cebbde6","Type":"ContainerStarted","Data":"91c7ba6626db87ef1f11154aae2521afdf3885c0072afebb22e5094ee79466ac"}
Nov 03 01:39:00.779175 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:00.779094    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb" podStartSLOduration=1.779071643 podStartE2EDuration="1.779071643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:00.77664845 +0000 UTC m=+5623.098185251" watchObservedRunningTime="2026-11-03 01:39:00.779071643 +0000 UTC m=+5623.100608444"
Nov 03 01:39:01.744577 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:01.744536    4550 generic.go:356] "Generic (PLEG): container finished" podID="978a4fec-a879-49ed-81e7-b1127cebbde6" containerID="9b7ba6b51f98de92dec7b18c615ff24b9d6a47882a26a38d516d3fb198bc24ac" exitCode=0
Nov 03 01:39:01.745041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:01.744624    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb" event={"ID":"978a4fec-a879-49ed-81e7-b1127cebbde6","Type":"ContainerDied","Data":"9b7ba6b51f98de92dec7b18c615ff24b9d6a47882a26a38d516d3fb198bc24ac"}
Nov 03 01:39:03.201860 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.200412    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:39:03.253650 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.253611    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca\") pod \"978a4fec-a879-49ed-81e7-b1127cebbde6\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") "
Nov 03 01:39:03.253942 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.253907    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca\") pod \"978a4fec-a879-49ed-81e7-b1127cebbde6\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") "
Nov 03 01:39:03.254223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.254204    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj\" (UniqueName: \"kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj\") pod \"978a4fec-a879-49ed-81e7-b1127cebbde6\" (UID: \"978a4fec-a879-49ed-81e7-b1127cebbde6\") "
Nov 03 01:39:03.255690 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.255667    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca" pod "978a4fec-a879-49ed-81e7-b1127cebbde6" (UID: "978a4fec-a879-49ed-81e7-b1127cebbde6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:03.259159 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.259114    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj" pod "978a4fec-a879-49ed-81e7-b1127cebbde6" (UID: "978a4fec-a879-49ed-81e7-b1127cebbde6"). InnerVolumeSpecName "kube-api-access-tbnnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:03.279107 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.279047    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca" pod "978a4fec-a879-49ed-81e7-b1127cebbde6" (UID: "978a4fec-a879-49ed-81e7-b1127cebbde6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:03.356766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.356715    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:03.356766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.356770    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978a4fec-a879-49ed-81e7-b1127cebbde6-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:03.356994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.356786    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbnnj\" (UniqueName: \"kubernetes.io/projected/978a4fec-a879-49ed-81e7-b1127cebbde6-kube-api-access-tbnnj\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:03.756176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.756141    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb"
Nov 03 01:39:03.756411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.756138    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0877206-zsfjb" event={"ID":"978a4fec-a879-49ed-81e7-b1127cebbde6","Type":"ContainerDied","Data":"91c7ba6626db87ef1f11154aae2521afdf3885c0072afebb22e5094ee79466ac"}
Nov 03 01:39:03.756411 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:03.756268    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91c7ba6626db87ef1f11154aae2521afdf3885c0072afebb22e5094ee79466ac"
Nov 03 01:39:18.883065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:18.883028    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:39:23.966240 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:23.966184    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:39:24.752664 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:39:24.752627    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:39:32.294468 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.294415    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"]
Nov 03 01:39:32.294994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.294810    4550 cpu_manager.go:414] "containerMap: removing container" podUID="978a4fec-a879-49ed-81e7-b1127cebbde6" containerName="revoke-keys"
Nov 03 01:39:32.294994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.294821    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978a4fec-a879-49ed-81e7-b1127cebbde6" containerName="revoke-keys"
Nov 03 01:39:32.294994 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.294902    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="978a4fec-a879-49ed-81e7-b1127cebbde6" containerName="revoke-keys"
Nov 03 01:39:32.298245 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.298228    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.317318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.317285    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"]
Nov 03 01:39:32.327697 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.327658    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:32.327899 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.327702    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:39:32.388542 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.388503    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.388737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.388565    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llhr\" (UniqueName: \"kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.388737 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.388683    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.489454 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.489389    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.489642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.489485    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.489642 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.489524    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llhr\" (UniqueName: \"kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.490062 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.490042    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.491927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.491899    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.509477 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.509425    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llhr\" (UniqueName: \"kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr\") pod \"maas-api-revoke-sub-f4cd3968-pp72g\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.611560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.611417    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:32.912575 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.912540    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"]
Nov 03 01:39:32.912575 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:39:32.912568    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ffcfd7_1b38_4895_8879_71b75efc1ab7.slice/crio-8a5a0db2d4f44b84d1c00927a2f111278e05f2878dfef9c2a57d03c2c4ecfb1e WatchSource:0}: Error finding container 8a5a0db2d4f44b84d1c00927a2f111278e05f2878dfef9c2a57d03c2c4ecfb1e: Status 404 returned error can't find the container with id 8a5a0db2d4f44b84d1c00927a2f111278e05f2878dfef9c2a57d03c2c4ecfb1e
Nov 03 01:39:32.965359 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:32.965317    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g" event={"ID":"64ffcfd7-1b38-4895-8879-71b75efc1ab7","Type":"ContainerStarted","Data":"8a5a0db2d4f44b84d1c00927a2f111278e05f2878dfef9c2a57d03c2c4ecfb1e"}
Nov 03 01:39:33.971867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:33.971828    4550 generic.go:356] "Generic (PLEG): container finished" podID="64ffcfd7-1b38-4895-8879-71b75efc1ab7" containerID="affa244202011aac6aaa53a4bc329d54a363b96bad039371f6da816d92f73517" exitCode=0
Nov 03 01:39:33.972278 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:33.971910    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g" event={"ID":"64ffcfd7-1b38-4895-8879-71b75efc1ab7","Type":"ContainerDied","Data":"affa244202011aac6aaa53a4bc329d54a363b96bad039371f6da816d92f73517"}
Nov 03 01:39:35.114335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.114301    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:35.218715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.218681    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca\") pod \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") "
Nov 03 01:39:35.218715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.218722    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr\" (UniqueName: \"kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr\") pod \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") "
Nov 03 01:39:35.218944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.218813    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca\") pod \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\" (UID: \"64ffcfd7-1b38-4895-8879-71b75efc1ab7\") "
Nov 03 01:39:35.219079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.219056    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca" pod "64ffcfd7-1b38-4895-8879-71b75efc1ab7" (UID: "64ffcfd7-1b38-4895-8879-71b75efc1ab7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:35.220885 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.220855    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca" pod "64ffcfd7-1b38-4895-8879-71b75efc1ab7" (UID: "64ffcfd7-1b38-4895-8879-71b75efc1ab7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:35.221065 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.220907    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr" pod "64ffcfd7-1b38-4895-8879-71b75efc1ab7" (UID: "64ffcfd7-1b38-4895-8879-71b75efc1ab7"). InnerVolumeSpecName "kube-api-access-2llhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:35.319776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.319674    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:35.319776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.319719    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64ffcfd7-1b38-4895-8879-71b75efc1ab7-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:35.319776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.319729    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llhr\" (UniqueName: \"kubernetes.io/projected/64ffcfd7-1b38-4895-8879-71b75efc1ab7-kube-api-access-2llhr\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:35.982776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.982738    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g" event={"ID":"64ffcfd7-1b38-4895-8879-71b75efc1ab7","Type":"ContainerDied","Data":"8a5a0db2d4f44b84d1c00927a2f111278e05f2878dfef9c2a57d03c2c4ecfb1e"}
Nov 03 01:39:35.982981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.982780    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4cd3968-pp72g"
Nov 03 01:39:35.982981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:35.982793    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5a0db2d4f44b84d1c00927a2f111278e05f2878dfef9c2a57d03c2c4ecfb1e"
Nov 03 01:39:37.703245 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:37.703210    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:39:42.652751 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.652708    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"]
Nov 03 01:39:42.653292 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.653275    4550 cpu_manager.go:414] "containerMap: removing container" podUID="64ffcfd7-1b38-4895-8879-71b75efc1ab7" containerName="revoke-keys"
Nov 03 01:39:42.653356 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.653295    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ffcfd7-1b38-4895-8879-71b75efc1ab7" containerName="revoke-keys"
Nov 03 01:39:42.653442 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.653420    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ffcfd7-1b38-4895-8879-71b75efc1ab7" containerName="revoke-keys"
Nov 03 01:39:42.656850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.656830    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.673079 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.673040    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"]
Nov 03 01:39:42.683177 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.683136    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:42.683363 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.683180    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:39:42.789201 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.789152    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfbt\" (UniqueName: \"kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.789458 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.789226    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.789458 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.789353    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.890140 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.890096    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.890397 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.890258    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.890397 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.890388    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfbt\" (UniqueName: \"kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.890989 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.890964    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.892808 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.892787    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.910940 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.910864    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfbt\" (UniqueName: \"kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt\") pod \"maas-api-revoke-sub-d755f8d3-5ltgq\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:42.971129 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:42.971088    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:43.271475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:43.271417    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"]
Nov 03 01:39:43.272776 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:39:43.272748    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b483c5_2338_4cbf_b399_9fcd344b3959.slice/crio-8a3e6318157e2c17ed52e290554ac279bde86382597785215f4478e757a03d21 WatchSource:0}: Error finding container 8a3e6318157e2c17ed52e290554ac279bde86382597785215f4478e757a03d21: Status 404 returned error can't find the container with id 8a3e6318157e2c17ed52e290554ac279bde86382597785215f4478e757a03d21
Nov 03 01:39:44.023214 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:44.023169    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq" event={"ID":"82b483c5-2338-4cbf-b399-9fcd344b3959","Type":"ContainerStarted","Data":"7b604e175161faae71b3a7e771b941a67049b1347a0cb5f2d5a4e5281344dcb2"}
Nov 03 01:39:44.023776 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:44.023226    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq" event={"ID":"82b483c5-2338-4cbf-b399-9fcd344b3959","Type":"ContainerStarted","Data":"8a3e6318157e2c17ed52e290554ac279bde86382597785215f4478e757a03d21"}
Nov 03 01:39:44.061925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:44.061872    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq" podStartSLOduration=2.061859179 podStartE2EDuration="2.061859179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:44.061311724 +0000 UTC m=+5666.382848549" watchObservedRunningTime="2026-11-03 01:39:44.061859179 +0000 UTC m=+5666.383395978"
Nov 03 01:39:45.033145 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:45.033101    4550 generic.go:356] "Generic (PLEG): container finished" podID="82b483c5-2338-4cbf-b399-9fcd344b3959" containerID="7b604e175161faae71b3a7e771b941a67049b1347a0cb5f2d5a4e5281344dcb2" exitCode=0
Nov 03 01:39:45.033657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:45.033164    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq" event={"ID":"82b483c5-2338-4cbf-b399-9fcd344b3959","Type":"ContainerDied","Data":"7b604e175161faae71b3a7e771b941a67049b1347a0cb5f2d5a4e5281344dcb2"}
Nov 03 01:39:46.175614 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.175583    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:46.323866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.323779    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt\" (UniqueName: \"kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt\") pod \"82b483c5-2338-4cbf-b399-9fcd344b3959\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") "
Nov 03 01:39:46.323866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.323822    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca\") pod \"82b483c5-2338-4cbf-b399-9fcd344b3959\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") "
Nov 03 01:39:46.324119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.323939    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca\") pod \"82b483c5-2338-4cbf-b399-9fcd344b3959\" (UID: \"82b483c5-2338-4cbf-b399-9fcd344b3959\") "
Nov 03 01:39:46.324272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.324246    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca" pod "82b483c5-2338-4cbf-b399-9fcd344b3959" (UID: "82b483c5-2338-4cbf-b399-9fcd344b3959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:46.325969 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.325942    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca" pod "82b483c5-2338-4cbf-b399-9fcd344b3959" (UID: "82b483c5-2338-4cbf-b399-9fcd344b3959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:46.326078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.325992    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt" pod "82b483c5-2338-4cbf-b399-9fcd344b3959" (UID: "82b483c5-2338-4cbf-b399-9fcd344b3959"). InnerVolumeSpecName "kube-api-access-nlfbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:46.425804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.425756    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfbt\" (UniqueName: \"kubernetes.io/projected/82b483c5-2338-4cbf-b399-9fcd344b3959-kube-api-access-nlfbt\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:46.425804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.425801    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:46.426041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:46.425818    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b483c5-2338-4cbf-b399-9fcd344b3959-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:47.043872 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:47.043843    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq"
Nov 03 01:39:47.044114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:47.043870    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d755f8d3-5ltgq" event={"ID":"82b483c5-2338-4cbf-b399-9fcd344b3959","Type":"ContainerDied","Data":"8a3e6318157e2c17ed52e290554ac279bde86382597785215f4478e757a03d21"}
Nov 03 01:39:47.044114 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:47.043927    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a3e6318157e2c17ed52e290554ac279bde86382597785215f4478e757a03d21"
Nov 03 01:39:52.992181 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:52.992145    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"]
Nov 03 01:39:52.992605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:52.992553    4550 cpu_manager.go:414] "containerMap: removing container" podUID="82b483c5-2338-4cbf-b399-9fcd344b3959" containerName="revoke-keys"
Nov 03 01:39:52.992605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:52.992566    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b483c5-2338-4cbf-b399-9fcd344b3959" containerName="revoke-keys"
Nov 03 01:39:52.992687 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:52.992640    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b483c5-2338-4cbf-b399-9fcd344b3959" containerName="revoke-keys"
Nov 03 01:39:52.995999 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:52.995975    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.013416 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.013378    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"]
Nov 03 01:39:53.023619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.023579    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:39:53.023839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.023631    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:39:53.099465 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.099407    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mlk\" (UniqueName: \"kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.099666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.099521    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.099666 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.099618    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.200835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.200798    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mlk\" (UniqueName: \"kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.201049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.200848    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.201049 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.200891    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.201614 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.201588    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.203284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.203260    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.220821 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.220794    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mlk\" (UniqueName: \"kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk\") pod \"maas-api-revoke-sub-f0d766b6-6bvvx\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.308579 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.308465    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:53.602415 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:53.602384    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"]
Nov 03 01:39:53.603890 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:39:53.603854    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6728ce4_0e1f_4b9f_a234_3fdaab0bfe72.slice/crio-006357a8dcfeb3961da575a20dbd0eb8d0c3515ecb4f780a976a9c1e38c16a4b WatchSource:0}: Error finding container 006357a8dcfeb3961da575a20dbd0eb8d0c3515ecb4f780a976a9c1e38c16a4b: Status 404 returned error can't find the container with id 006357a8dcfeb3961da575a20dbd0eb8d0c3515ecb4f780a976a9c1e38c16a4b
Nov 03 01:39:54.079295 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:54.079242    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx" event={"ID":"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72","Type":"ContainerStarted","Data":"1cd384e44e8d8b5343271cf6aad0d4ac46cda88af30b63137c0567194103fcd4"}
Nov 03 01:39:54.079835 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:54.079315    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx" event={"ID":"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72","Type":"ContainerStarted","Data":"006357a8dcfeb3961da575a20dbd0eb8d0c3515ecb4f780a976a9c1e38c16a4b"}
Nov 03 01:39:54.120153 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:54.120091    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx" podStartSLOduration=2.120076333 podStartE2EDuration="2.120076333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:54.117997373 +0000 UTC m=+5676.439534190" watchObservedRunningTime="2026-11-03 01:39:54.120076333 +0000 UTC m=+5676.441613131"
Nov 03 01:39:54.723414 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:39:54.723375    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:39:55.086497 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:55.086404    4550 generic.go:356] "Generic (PLEG): container finished" podID="b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" containerID="1cd384e44e8d8b5343271cf6aad0d4ac46cda88af30b63137c0567194103fcd4" exitCode=0
Nov 03 01:39:55.086888 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:55.086490    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx" event={"ID":"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72","Type":"ContainerDied","Data":"1cd384e44e8d8b5343271cf6aad0d4ac46cda88af30b63137c0567194103fcd4"}
Nov 03 01:39:56.227017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.226989    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:39:56.329824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.329790    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca\") pod \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") "
Nov 03 01:39:56.330058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.329839    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca\") pod \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") "
Nov 03 01:39:56.330058 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.329894    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk\" (UniqueName: \"kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk\") pod \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\" (UID: \"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72\") "
Nov 03 01:39:56.330223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.330193    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca" pod "b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" (UID: "b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:39:56.332189 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.332162    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca" pod "b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" (UID: "b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:39:56.332335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.332248    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk" pod "b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" (UID: "b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72"). InnerVolumeSpecName "kube-api-access-95mlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:39:56.431647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.431591    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:56.431647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.431642    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mlk\" (UniqueName: \"kubernetes.io/projected/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-kube-api-access-95mlk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:56.431647 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:56.431655    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:39:57.097734 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:57.097697    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx" event={"ID":"b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72","Type":"ContainerDied","Data":"006357a8dcfeb3961da575a20dbd0eb8d0c3515ecb4f780a976a9c1e38c16a4b"}
Nov 03 01:39:57.097905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:57.097752    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006357a8dcfeb3961da575a20dbd0eb8d0c3515ecb4f780a976a9c1e38c16a4b"
Nov 03 01:39:57.097905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:39:57.097714    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0d766b6-6bvvx"
Nov 03 01:40:03.353512 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.353471    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"]
Nov 03 01:40:03.354153 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.354024    4550 cpu_manager.go:414] "containerMap: removing container" podUID="b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" containerName="revoke-keys"
Nov 03 01:40:03.354153 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.354046    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" containerName="revoke-keys"
Nov 03 01:40:03.354284 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.354172    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72" containerName="revoke-keys"
Nov 03 01:40:03.357708 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.357683    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.374914 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.374883    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"]
Nov 03 01:40:03.385559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.385531    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:03.385848 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.385596    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:40:03.398919 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.398879    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.399073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.398991    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.399073 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.399018    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdq6\" (UniqueName: \"kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.499957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.499904    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.499957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.499971    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.500268 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.499989    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdq6\" (UniqueName: \"kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.500722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.500696    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.502603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.502568    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.521196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.521162    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdq6\" (UniqueName: \"kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6\") pod \"maas-api-revoke-sub-7c759a18-6n4hd\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.671288 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.671247    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:03.978375 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:40:03.978315    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ed6a4f_2201_46f7_aea3_930a0cbd4bd6.slice/crio-dc8838c1e5e6568e69bb557589e6808e51f554182a7f21eece72ed218eead91a WatchSource:0}: Error finding container dc8838c1e5e6568e69bb557589e6808e51f554182a7f21eece72ed218eead91a: Status 404 returned error can't find the container with id dc8838c1e5e6568e69bb557589e6808e51f554182a7f21eece72ed218eead91a
Nov 03 01:40:03.979236 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:03.979206    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"]
Nov 03 01:40:04.131517 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:04.131480    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd" event={"ID":"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6","Type":"ContainerStarted","Data":"dc8838c1e5e6568e69bb557589e6808e51f554182a7f21eece72ed218eead91a"}
Nov 03 01:40:05.140452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:05.140338    4550 generic.go:356] "Generic (PLEG): container finished" podID="a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" containerID="8a183eb213ef9cf9aa3252903bf28bc64acb3044c6fc9791c142a03a729e0ae5" exitCode=0
Nov 03 01:40:05.140452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:05.140401    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd" event={"ID":"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6","Type":"ContainerDied","Data":"8a183eb213ef9cf9aa3252903bf28bc64acb3044c6fc9791c142a03a729e0ae5"}
Nov 03 01:40:06.561244 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.560892    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:06.650249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.650210    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca\") pod \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") "
Nov 03 01:40:06.650498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.650350    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca\") pod \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") "
Nov 03 01:40:06.650498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.650395    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6\" (UniqueName: \"kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6\") pod \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\" (UID: \"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6\") "
Nov 03 01:40:06.650722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.650689    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca" pod "a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" (UID: "a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:06.652564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.652535    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca" pod "a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" (UID: "a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:06.652665 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.652611    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6" pod "a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" (UID: "a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6"). InnerVolumeSpecName "kube-api-access-rxdq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:06.751875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.751834    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:06.751875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.751872    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:06.751875 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:06.751887    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdq6\" (UniqueName: \"kubernetes.io/projected/a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6-kube-api-access-rxdq6\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:07.153785 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:07.153752    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd"
Nov 03 01:40:07.153981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:07.153756    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c759a18-6n4hd" event={"ID":"a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6","Type":"ContainerDied","Data":"dc8838c1e5e6568e69bb557589e6808e51f554182a7f21eece72ed218eead91a"}
Nov 03 01:40:07.153981 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:07.153876    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc8838c1e5e6568e69bb557589e6808e51f554182a7f21eece72ed218eead91a"
Nov 03 01:40:13.722169 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.722128    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"]
Nov 03 01:40:13.722721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.722564    4550 cpu_manager.go:414] "containerMap: removing container" podUID="a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" containerName="revoke-keys"
Nov 03 01:40:13.722721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.722577    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" containerName="revoke-keys"
Nov 03 01:40:13.722721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.722639    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6" containerName="revoke-keys"
Nov 03 01:40:13.725716 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.725687    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.744119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.744080    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"]
Nov 03 01:40:13.754597 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.754562    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:13.754801 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.754618    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:40:13.822504 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.822468    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.822703 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.822529    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.826471 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.822994    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q56k\" (UniqueName: \"kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.923923 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.923873    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.924166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.923957    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.924166 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.924005    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q56k\" (UniqueName: \"kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.924792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.924767    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.926318 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.926294    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:13.944420 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:13.944392    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q56k\" (UniqueName: \"kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k\") pod \"maas-api-revoke-sub-c3cc145c-kpz9l\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:14.045499 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:14.045401    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:14.339773 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:40:14.339742    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b5a37a_40c4_4f8b_917d_383d4cab4c3f.slice/crio-ebdc41c5c64dd8ac6170f227ba354e053717907c36b1f8e686b50ff53a29b522 WatchSource:0}: Error finding container ebdc41c5c64dd8ac6170f227ba354e053717907c36b1f8e686b50ff53a29b522: Status 404 returned error can't find the container with id ebdc41c5c64dd8ac6170f227ba354e053717907c36b1f8e686b50ff53a29b522
Nov 03 01:40:14.340306 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:14.340283    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"]
Nov 03 01:40:15.195569 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:15.195523    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l" event={"ID":"27b5a37a-40c4-4f8b-917d-383d4cab4c3f","Type":"ContainerStarted","Data":"e180190b813fd4c1d0f0bc910666b72728a7bf03635305bc129ad26121d94d46"}
Nov 03 01:40:15.196195 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:15.195599    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l" event={"ID":"27b5a37a-40c4-4f8b-917d-383d4cab4c3f","Type":"ContainerStarted","Data":"ebdc41c5c64dd8ac6170f227ba354e053717907c36b1f8e686b50ff53a29b522"}
Nov 03 01:40:15.238196 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:15.238122    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l" podStartSLOduration=2.238100684 podStartE2EDuration="2.238100684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:15.236786234 +0000 UTC m=+5697.558323034" watchObservedRunningTime="2026-11-03 01:40:15.238100684 +0000 UTC m=+5697.559637483"
Nov 03 01:40:16.217319 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:16.217278    4550 generic.go:356] "Generic (PLEG): container finished" podID="27b5a37a-40c4-4f8b-917d-383d4cab4c3f" containerID="e180190b813fd4c1d0f0bc910666b72728a7bf03635305bc129ad26121d94d46" exitCode=0
Nov 03 01:40:16.218470 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:16.218264    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l" event={"ID":"27b5a37a-40c4-4f8b-917d-383d4cab4c3f","Type":"ContainerDied","Data":"e180190b813fd4c1d0f0bc910666b72728a7bf03635305bc129ad26121d94d46"}
Nov 03 01:40:17.653156 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.652846    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:17.826605 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.826570    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k\" (UniqueName: \"kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k\") pod \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") "
Nov 03 01:40:17.826814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.826638    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca\") pod \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") "
Nov 03 01:40:17.826814 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.826666    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca\") pod \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\" (UID: \"27b5a37a-40c4-4f8b-917d-383d4cab4c3f\") "
Nov 03 01:40:17.827083 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.827059    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca" pod "27b5a37a-40c4-4f8b-917d-383d4cab4c3f" (UID: "27b5a37a-40c4-4f8b-917d-383d4cab4c3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:17.828825 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.828799    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca" pod "27b5a37a-40c4-4f8b-917d-383d4cab4c3f" (UID: "27b5a37a-40c4-4f8b-917d-383d4cab4c3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:17.828941 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.828917    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k" pod "27b5a37a-40c4-4f8b-917d-383d4cab4c3f" (UID: "27b5a37a-40c4-4f8b-917d-383d4cab4c3f"). InnerVolumeSpecName "kube-api-access-6q56k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:17.927505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.927470    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q56k\" (UniqueName: \"kubernetes.io/projected/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-kube-api-access-6q56k\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:17.927505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.927499    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:17.927505 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:17.927510    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27b5a37a-40c4-4f8b-917d-383d4cab4c3f-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:18.236353 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:18.236268    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l"
Nov 03 01:40:18.236536 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:18.236274    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cc145c-kpz9l" event={"ID":"27b5a37a-40c4-4f8b-917d-383d4cab4c3f","Type":"ContainerDied","Data":"ebdc41c5c64dd8ac6170f227ba354e053717907c36b1f8e686b50ff53a29b522"}
Nov 03 01:40:18.236536 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:18.236399    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebdc41c5c64dd8ac6170f227ba354e053717907c36b1f8e686b50ff53a29b522"
Nov 03 01:40:18.990291 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:18.990242    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:40:24.080570 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.080523    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:40:24.149258 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.149223    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"]
Nov 03 01:40:24.149623 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.149608    4550 cpu_manager.go:414] "containerMap: removing container" podUID="27b5a37a-40c4-4f8b-917d-383d4cab4c3f" containerName="revoke-keys"
Nov 03 01:40:24.149682 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.149624    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27b5a37a-40c4-4f8b-917d-383d4cab4c3f" containerName="revoke-keys"
Nov 03 01:40:24.149767 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.149749    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="27b5a37a-40c4-4f8b-917d-383d4cab4c3f" containerName="revoke-keys"
Nov 03 01:40:24.153088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.153063    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.171889 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.171854    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"]
Nov 03 01:40:24.181845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.181808    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:24.182043 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.181876    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:40:24.284232 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.284191    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.284232 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.284236    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.284571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.284368    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshp4\" (UniqueName: \"kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.385580 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.385473    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.385786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.385613    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.385786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.385659    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshp4\" (UniqueName: \"kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.386274 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.386251    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.387957 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.387932    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.405675 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.405641    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshp4\" (UniqueName: \"kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4\") pod \"maas-api-revoke-sub-fcab913e-zkwjx\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.479853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.479822    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:24.722890 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:40:24.722846    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:40:24.780039 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:24.780008    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"]
Nov 03 01:40:24.780557 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:40:24.780527    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168151b5_24de_42c3_bc11_5a08d6a252af.slice/crio-f92fa23478c7b6cd81ff9b0300154c46fc008556c50f62f79200a6639a1f5a1d WatchSource:0}: Error finding container f92fa23478c7b6cd81ff9b0300154c46fc008556c50f62f79200a6639a1f5a1d: Status 404 returned error can't find the container with id f92fa23478c7b6cd81ff9b0300154c46fc008556c50f62f79200a6639a1f5a1d
Nov 03 01:40:25.271766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:25.271669    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx" event={"ID":"168151b5-24de-42c3-bc11-5a08d6a252af","Type":"ContainerStarted","Data":"b717aeac80d5bea225c75fa629f67a0ce20536957fb9c59c1705b4d49c794c0b"}
Nov 03 01:40:25.271766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:25.271728    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx" event={"ID":"168151b5-24de-42c3-bc11-5a08d6a252af","Type":"ContainerStarted","Data":"f92fa23478c7b6cd81ff9b0300154c46fc008556c50f62f79200a6639a1f5a1d"}
Nov 03 01:40:25.312012 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:25.311949    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx" podStartSLOduration=1.311936256 podStartE2EDuration="1.311936256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:25.310910442 +0000 UTC m=+5707.632447242" watchObservedRunningTime="2026-11-03 01:40:25.311936256 +0000 UTC m=+5707.633473054"
Nov 03 01:40:26.281255 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:26.281211    4550 generic.go:356] "Generic (PLEG): container finished" podID="168151b5-24de-42c3-bc11-5a08d6a252af" containerID="b717aeac80d5bea225c75fa629f67a0ce20536957fb9c59c1705b4d49c794c0b" exitCode=0
Nov 03 01:40:26.281824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:26.281367    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx" event={"ID":"168151b5-24de-42c3-bc11-5a08d6a252af","Type":"ContainerDied","Data":"b717aeac80d5bea225c75fa629f67a0ce20536957fb9c59c1705b4d49c794c0b"}
Nov 03 01:40:27.583639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.569699    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-rcjbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:27.583639 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.569766    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-rcjbb" podUID="caac059c-fc5f-4b53-99d5-c98bccf3480d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:27.655530 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.655504    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:27.739838 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.739790    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca\") pod \"168151b5-24de-42c3-bc11-5a08d6a252af\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") "
Nov 03 01:40:27.740048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.739869    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4\" (UniqueName: \"kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4\") pod \"168151b5-24de-42c3-bc11-5a08d6a252af\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") "
Nov 03 01:40:27.740048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.739939    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca\") pod \"168151b5-24de-42c3-bc11-5a08d6a252af\" (UID: \"168151b5-24de-42c3-bc11-5a08d6a252af\") "
Nov 03 01:40:27.740409 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.740357    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca" pod "168151b5-24de-42c3-bc11-5a08d6a252af" (UID: "168151b5-24de-42c3-bc11-5a08d6a252af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:27.742340 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.742304    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4" pod "168151b5-24de-42c3-bc11-5a08d6a252af" (UID: "168151b5-24de-42c3-bc11-5a08d6a252af"). InnerVolumeSpecName "kube-api-access-pshp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:27.742648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.742630    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca" pod "168151b5-24de-42c3-bc11-5a08d6a252af" (UID: "168151b5-24de-42c3-bc11-5a08d6a252af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:27.840850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.840760    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:27.840850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.840793    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/168151b5-24de-42c3-bc11-5a08d6a252af-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:27.840850 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:27.840810    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pshp4\" (UniqueName: \"kubernetes.io/projected/168151b5-24de-42c3-bc11-5a08d6a252af-kube-api-access-pshp4\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:28.306263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:28.306230    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx"
Nov 03 01:40:28.306263 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:28.306251    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fcab913e-zkwjx" event={"ID":"168151b5-24de-42c3-bc11-5a08d6a252af","Type":"ContainerDied","Data":"f92fa23478c7b6cd81ff9b0300154c46fc008556c50f62f79200a6639a1f5a1d"}
Nov 03 01:40:28.306562 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:28.306293    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f92fa23478c7b6cd81ff9b0300154c46fc008556c50f62f79200a6639a1f5a1d"
Nov 03 01:40:34.496227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.496130    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"]
Nov 03 01:40:34.496870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.496668    4550 cpu_manager.go:414] "containerMap: removing container" podUID="168151b5-24de-42c3-bc11-5a08d6a252af" containerName="revoke-keys"
Nov 03 01:40:34.496870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.496687    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168151b5-24de-42c3-bc11-5a08d6a252af" containerName="revoke-keys"
Nov 03 01:40:34.496870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.496811    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="168151b5-24de-42c3-bc11-5a08d6a252af" containerName="revoke-keys"
Nov 03 01:40:34.500629 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.500599    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.518610 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.518580    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"]
Nov 03 01:40:34.528805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.528770    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:34.529018 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.528879    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:40:34.594836 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.594804    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qdk\" (UniqueName: \"kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.595026 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.594866    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.595026 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.594999    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.695955 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.695908    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.696139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.695989    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qdk\" (UniqueName: \"kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.696139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.696052    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.696899 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.696871    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.698252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.698231    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.715947 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.715921    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qdk\" (UniqueName: \"kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk\") pod \"maas-api-revoke-sub-574e406d-grh79\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:34.814473 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:34.814356    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:35.111801 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:40:35.111753    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4db3884_6598_45f1_ab96_d86816dd4a95.slice/crio-2ec33181c26dc8f87477539fe0e108712751f69913db48c22b3073157162d820 WatchSource:0}: Error finding container 2ec33181c26dc8f87477539fe0e108712751f69913db48c22b3073157162d820: Status 404 returned error can't find the container with id 2ec33181c26dc8f87477539fe0e108712751f69913db48c22b3073157162d820
Nov 03 01:40:35.111986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:35.111960    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"]
Nov 03 01:40:35.340619 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:35.340581    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79" event={"ID":"f4db3884-6598-45f1-ab96-d86816dd4a95","Type":"ContainerStarted","Data":"2ec33181c26dc8f87477539fe0e108712751f69913db48c22b3073157162d820"}
Nov 03 01:40:36.349457 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:36.349348    4550 generic.go:356] "Generic (PLEG): container finished" podID="f4db3884-6598-45f1-ab96-d86816dd4a95" containerID="4ded5fcf85bbaf19c7953356ae70989c70b3743191b37aee7331744dc6f5572a" exitCode=0
Nov 03 01:40:36.349851 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:36.349464    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79" event={"ID":"f4db3884-6598-45f1-ab96-d86816dd4a95","Type":"ContainerDied","Data":"4ded5fcf85bbaf19c7953356ae70989c70b3743191b37aee7331744dc6f5572a"}
Nov 03 01:40:37.754820 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.754794    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:37.842905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.842873    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk\" (UniqueName: \"kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk\") pod \"f4db3884-6598-45f1-ab96-d86816dd4a95\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") "
Nov 03 01:40:37.843116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.843019    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca\") pod \"f4db3884-6598-45f1-ab96-d86816dd4a95\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") "
Nov 03 01:40:37.843116 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.843059    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca\") pod \"f4db3884-6598-45f1-ab96-d86816dd4a95\" (UID: \"f4db3884-6598-45f1-ab96-d86816dd4a95\") "
Nov 03 01:40:37.843670 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.843645    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca" pod "f4db3884-6598-45f1-ab96-d86816dd4a95" (UID: "f4db3884-6598-45f1-ab96-d86816dd4a95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:37.845099 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.845071    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk" pod "f4db3884-6598-45f1-ab96-d86816dd4a95" (UID: "f4db3884-6598-45f1-ab96-d86816dd4a95"). InnerVolumeSpecName "kube-api-access-j9qdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:37.845356 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.845339    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca" pod "f4db3884-6598-45f1-ab96-d86816dd4a95" (UID: "f4db3884-6598-45f1-ab96-d86816dd4a95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:37.944091 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.944055    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qdk\" (UniqueName: \"kubernetes.io/projected/f4db3884-6598-45f1-ab96-d86816dd4a95-kube-api-access-j9qdk\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:37.944091 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.944084    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:37.944091 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:37.944094    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4db3884-6598-45f1-ab96-d86816dd4a95-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:38.263455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:38.262551    4550 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-gknbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:38.263455 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:38.262624    4550 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-gknbx" podUID="f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:38.368609 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:38.368568    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79" event={"ID":"f4db3884-6598-45f1-ab96-d86816dd4a95","Type":"ContainerDied","Data":"2ec33181c26dc8f87477539fe0e108712751f69913db48c22b3073157162d820"}
Nov 03 01:40:38.368925 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:38.368904    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec33181c26dc8f87477539fe0e108712751f69913db48c22b3073157162d820"
Nov 03 01:40:38.369121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:38.369106    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574e406d-grh79"
Nov 03 01:40:44.847800 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.847768    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"]
Nov 03 01:40:44.848246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.848154    4550 cpu_manager.go:414] "containerMap: removing container" podUID="f4db3884-6598-45f1-ab96-d86816dd4a95" containerName="revoke-keys"
Nov 03 01:40:44.848246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.848166    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4db3884-6598-45f1-ab96-d86816dd4a95" containerName="revoke-keys"
Nov 03 01:40:44.848246 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.848240    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4db3884-6598-45f1-ab96-d86816dd4a95" containerName="revoke-keys"
Nov 03 01:40:44.851559 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.851537    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:44.880170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.880138    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:44.880368 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.880234    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:40:44.884008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.883984    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"]
Nov 03 01:40:44.906571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.906534    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:44.906571 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.906573    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqckq\" (UniqueName: \"kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:44.906884 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:44.906695    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.007899 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.007846    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.008138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.007908    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqckq\" (UniqueName: \"kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.008138 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.007981    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.008742 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.008713    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.010395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.010347    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.027840 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.027812    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqckq\" (UniqueName: \"kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq\") pod \"maas-api-revoke-sub-64c806ff-hkmnw\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.169867 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.169831    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:45.464947 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:40:45.464911    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74164494_9b5d_496e_84a0_369a51087d1a.slice/crio-1fa5dac2397b1cbaae27916f4c0a1bae5896dd03d1cb9307fb58f9db7497ea51 WatchSource:0}: Error finding container 1fa5dac2397b1cbaae27916f4c0a1bae5896dd03d1cb9307fb58f9db7497ea51: Status 404 returned error can't find the container with id 1fa5dac2397b1cbaae27916f4c0a1bae5896dd03d1cb9307fb58f9db7497ea51
Nov 03 01:40:45.465496 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:45.465464    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"]
Nov 03 01:40:46.411511 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:46.411461    4550 generic.go:356] "Generic (PLEG): container finished" podID="74164494-9b5d-496e-84a0-369a51087d1a" containerID="1ae5e95adbc65e4f33ee171e7bd12cbf816723e9cff4296f8178dd9cd3a5499e" exitCode=0
Nov 03 01:40:46.412067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:46.411535    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw" event={"ID":"74164494-9b5d-496e-84a0-369a51087d1a","Type":"ContainerDied","Data":"1ae5e95adbc65e4f33ee171e7bd12cbf816723e9cff4296f8178dd9cd3a5499e"}
Nov 03 01:40:46.412067 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:46.411602    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw" event={"ID":"74164494-9b5d-496e-84a0-369a51087d1a","Type":"ContainerStarted","Data":"1fa5dac2397b1cbaae27916f4c0a1bae5896dd03d1cb9307fb58f9db7497ea51"}
Nov 03 01:40:47.966890 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:47.966855    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:48.052544 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.052504    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca\") pod \"74164494-9b5d-496e-84a0-369a51087d1a\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") "
Nov 03 01:40:48.052773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.052592    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq\" (UniqueName: \"kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq\") pod \"74164494-9b5d-496e-84a0-369a51087d1a\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") "
Nov 03 01:40:48.052773 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.052641    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca\") pod \"74164494-9b5d-496e-84a0-369a51087d1a\" (UID: \"74164494-9b5d-496e-84a0-369a51087d1a\") "
Nov 03 01:40:48.053006 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.052981    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca" pod "74164494-9b5d-496e-84a0-369a51087d1a" (UID: "74164494-9b5d-496e-84a0-369a51087d1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:48.054634 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.054611    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq" pod "74164494-9b5d-496e-84a0-369a51087d1a" (UID: "74164494-9b5d-496e-84a0-369a51087d1a"). InnerVolumeSpecName "kube-api-access-kqckq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:48.054714 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.054695    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca" pod "74164494-9b5d-496e-84a0-369a51087d1a" (UID: "74164494-9b5d-496e-84a0-369a51087d1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:48.154452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.154393    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqckq\" (UniqueName: \"kubernetes.io/projected/74164494-9b5d-496e-84a0-369a51087d1a-kube-api-access-kqckq\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:48.154452 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.154426    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:48.154704 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.154472    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74164494-9b5d-496e-84a0-369a51087d1a-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:48.442796 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.442755    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw" event={"ID":"74164494-9b5d-496e-84a0-369a51087d1a","Type":"ContainerDied","Data":"1fa5dac2397b1cbaae27916f4c0a1bae5896dd03d1cb9307fb58f9db7497ea51"}
Nov 03 01:40:48.442996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.442819    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c806ff-hkmnw"
Nov 03 01:40:48.442996 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:48.442837    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fa5dac2397b1cbaae27916f4c0a1bae5896dd03d1cb9307fb58f9db7497ea51"
Nov 03 01:40:54.722886 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:40:54.722850    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:40:55.208170 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.208135    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"]
Nov 03 01:40:55.208561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.208545    4550 cpu_manager.go:414] "containerMap: removing container" podUID="74164494-9b5d-496e-84a0-369a51087d1a" containerName="revoke-keys"
Nov 03 01:40:55.208561 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.208560    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74164494-9b5d-496e-84a0-369a51087d1a" containerName="revoke-keys"
Nov 03 01:40:55.208722 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.208631    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="74164494-9b5d-496e-84a0-369a51087d1a" containerName="revoke-keys"
Nov 03 01:40:55.211973 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.211949    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.231188 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.231154    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"]
Nov 03 01:40:55.240883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.240851    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:40:55.240883 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.240852    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:40:55.355148 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.355107    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tj8k\" (UniqueName: \"kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.355342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.355166    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.355342 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.355273    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.456040 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.455997    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.456249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.456051    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.456249 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.456148    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tj8k\" (UniqueName: \"kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.456706 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.456682    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.458564 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.458509    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.475986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.475960    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tj8k\" (UniqueName: \"kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k\") pod \"maas-api-revoke-sub-bf03952d-spjdd\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.525563 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.525534    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:55.820878 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:40:55.820847    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5b8e45_58d2_4607_9041_6ae81abf1715.slice/crio-bf44d0bae61e0914df1fe3f42a09172d2adb07adbfc702a5c615a3794b176322 WatchSource:0}: Error finding container bf44d0bae61e0914df1fe3f42a09172d2adb07adbfc702a5c615a3794b176322: Status 404 returned error can't find the container with id bf44d0bae61e0914df1fe3f42a09172d2adb07adbfc702a5c615a3794b176322
Nov 03 01:40:55.823216 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:55.823130    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"]
Nov 03 01:40:56.486911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:56.486871    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd" event={"ID":"2f5b8e45-58d2-4607-9041-6ae81abf1715","Type":"ContainerStarted","Data":"1405d53869a5c387937da63b3f546d94a9532b0ec781cb90999a49a20d5c4b3e"}
Nov 03 01:40:56.487167 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:56.486938    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd" event={"ID":"2f5b8e45-58d2-4607-9041-6ae81abf1715","Type":"ContainerStarted","Data":"bf44d0bae61e0914df1fe3f42a09172d2adb07adbfc702a5c615a3794b176322"}
Nov 03 01:40:56.528627 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:56.528572    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd" podStartSLOduration=1.528555364 podStartE2EDuration="1.528555364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:56.527480381 +0000 UTC m=+5738.849017210" watchObservedRunningTime="2026-11-03 01:40:56.528555364 +0000 UTC m=+5738.850092163"
Nov 03 01:40:57.513462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:57.513019    4550 generic.go:356] "Generic (PLEG): container finished" podID="2f5b8e45-58d2-4607-9041-6ae81abf1715" containerID="1405d53869a5c387937da63b3f546d94a9532b0ec781cb90999a49a20d5c4b3e" exitCode=0
Nov 03 01:40:57.513462 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:57.513083    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd" event={"ID":"2f5b8e45-58d2-4607-9041-6ae81abf1715","Type":"ContainerDied","Data":"1405d53869a5c387937da63b3f546d94a9532b0ec781cb90999a49a20d5c4b3e"}
Nov 03 01:40:58.988094 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:58.987697    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:59.119105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.119052    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca\") pod \"2f5b8e45-58d2-4607-9041-6ae81abf1715\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") "
Nov 03 01:40:59.119367 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.119125    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k\" (UniqueName: \"kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k\") pod \"2f5b8e45-58d2-4607-9041-6ae81abf1715\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") "
Nov 03 01:40:59.119367 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.119195    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca\") pod \"2f5b8e45-58d2-4607-9041-6ae81abf1715\" (UID: \"2f5b8e45-58d2-4607-9041-6ae81abf1715\") "
Nov 03 01:40:59.120109 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.120039    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca" pod "2f5b8e45-58d2-4607-9041-6ae81abf1715" (UID: "2f5b8e45-58d2-4607-9041-6ae81abf1715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:40:59.123721 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.123686    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca" pod "2f5b8e45-58d2-4607-9041-6ae81abf1715" (UID: "2f5b8e45-58d2-4607-9041-6ae81abf1715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:40:59.124223 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.124178    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k" pod "2f5b8e45-58d2-4607-9041-6ae81abf1715" (UID: "2f5b8e45-58d2-4607-9041-6ae81abf1715"). InnerVolumeSpecName "kube-api-access-4tj8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:40:59.220938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.220839    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:59.220938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.220878    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f5b8e45-58d2-4607-9041-6ae81abf1715-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:59.220938 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.220888    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tj8k\" (UniqueName: \"kubernetes.io/projected/2f5b8e45-58d2-4607-9041-6ae81abf1715-kube-api-access-4tj8k\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:40:59.533176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.533080    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd" event={"ID":"2f5b8e45-58d2-4607-9041-6ae81abf1715","Type":"ContainerDied","Data":"bf44d0bae61e0914df1fe3f42a09172d2adb07adbfc702a5c615a3794b176322"}
Nov 03 01:40:59.533176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.533130    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf03952d-spjdd"
Nov 03 01:40:59.533401 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:40:59.533134    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf44d0bae61e0914df1fe3f42a09172d2adb07adbfc702a5c615a3794b176322"
Nov 03 01:41:05.567176 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.567138    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"]
Nov 03 01:41:05.567786 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.567765    4550 cpu_manager.go:414] "containerMap: removing container" podUID="2f5b8e45-58d2-4607-9041-6ae81abf1715" containerName="revoke-keys"
Nov 03 01:41:05.567839 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.567790    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5b8e45-58d2-4607-9041-6ae81abf1715" containerName="revoke-keys"
Nov 03 01:41:05.567904 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.567891    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5b8e45-58d2-4607-9041-6ae81abf1715" containerName="revoke-keys"
Nov 03 01:41:05.570794 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.570767    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.591899 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.591861    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"]
Nov 03 01:41:05.600387 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.600355    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:41:05.600626 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.600362    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:41:05.681965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.681928    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.681965 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.681969    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wz4l\" (UniqueName: \"kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.682250 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.682122    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.782855 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.782819    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.783056 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.782870    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.783056 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.782894    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wz4l\" (UniqueName: \"kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.783519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.783495    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.785412 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.785387    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.802501 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.802475    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wz4l\" (UniqueName: \"kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l\") pod \"maas-api-revoke-sub-a51b5ee1-d7kzm\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:05.884089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:05.883991    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:06.179373 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:06.179331    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"]
Nov 03 01:41:06.180337 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:41:06.180296    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d418fc_d568_4403_8fb4_7d98651ebdff.slice/crio-76f6f63b3a3093eca16b931bc081affe53be801e0c6565f2a221d4ebcd18227d WatchSource:0}: Error finding container 76f6f63b3a3093eca16b931bc081affe53be801e0c6565f2a221d4ebcd18227d: Status 404 returned error can't find the container with id 76f6f63b3a3093eca16b931bc081affe53be801e0c6565f2a221d4ebcd18227d
Nov 03 01:41:06.571804 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:06.571761    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm" event={"ID":"61d418fc-d568-4403-8fb4-7d98651ebdff","Type":"ContainerStarted","Data":"08215ddc2bd0e4230e3577c739a2e3922aa9f7006fabed34792f85466a3a926c"}
Nov 03 01:41:06.572221 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:06.571821    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm" event={"ID":"61d418fc-d568-4403-8fb4-7d98651ebdff","Type":"ContainerStarted","Data":"76f6f63b3a3093eca16b931bc081affe53be801e0c6565f2a221d4ebcd18227d"}
Nov 03 01:41:06.612107 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:06.612050    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm" podStartSLOduration=1.612036077 podStartE2EDuration="1.612036077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:06.611776716 +0000 UTC m=+5748.933313533" watchObservedRunningTime="2026-11-03 01:41:06.612036077 +0000 UTC m=+5748.933572875"
Nov 03 01:41:07.583791 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:07.583750    4550 generic.go:356] "Generic (PLEG): container finished" podID="61d418fc-d568-4403-8fb4-7d98651ebdff" containerID="08215ddc2bd0e4230e3577c739a2e3922aa9f7006fabed34792f85466a3a926c" exitCode=0
Nov 03 01:41:07.584350 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:07.583931    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm" event={"ID":"61d418fc-d568-4403-8fb4-7d98651ebdff","Type":"ContainerDied","Data":"08215ddc2bd0e4230e3577c739a2e3922aa9f7006fabed34792f85466a3a926c"}
Nov 03 01:41:09.041887 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.041856    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:09.237905 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.237774    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l\" (UniqueName: \"kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l\") pod \"61d418fc-d568-4403-8fb4-7d98651ebdff\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") "
Nov 03 01:41:09.238121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.237924    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca\") pod \"61d418fc-d568-4403-8fb4-7d98651ebdff\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") "
Nov 03 01:41:09.238121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.237983    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca\") pod \"61d418fc-d568-4403-8fb4-7d98651ebdff\" (UID: \"61d418fc-d568-4403-8fb4-7d98651ebdff\") "
Nov 03 01:41:09.238743 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.238713    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca" pod "61d418fc-d568-4403-8fb4-7d98651ebdff" (UID: "61d418fc-d568-4403-8fb4-7d98651ebdff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:09.241696 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.241662    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l" pod "61d418fc-d568-4403-8fb4-7d98651ebdff" (UID: "61d418fc-d568-4403-8fb4-7d98651ebdff"). InnerVolumeSpecName "kube-api-access-4wz4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:09.242210 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.242184    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca" pod "61d418fc-d568-4403-8fb4-7d98651ebdff" (UID: "61d418fc-d568-4403-8fb4-7d98651ebdff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:09.338810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.338773    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:09.338810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.338803    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d418fc-d568-4403-8fb4-7d98651ebdff-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:09.338810 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.338814    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wz4l\" (UniqueName: \"kubernetes.io/projected/61d418fc-d568-4403-8fb4-7d98651ebdff-kube-api-access-4wz4l\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:09.613149 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.613069    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm"
Nov 03 01:41:09.613149 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.613092    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a51b5ee1-d7kzm" event={"ID":"61d418fc-d568-4403-8fb4-7d98651ebdff","Type":"ContainerDied","Data":"76f6f63b3a3093eca16b931bc081affe53be801e0c6565f2a221d4ebcd18227d"}
Nov 03 01:41:09.613149 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:09.613138    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76f6f63b3a3093eca16b931bc081affe53be801e0c6565f2a221d4ebcd18227d"
Nov 03 01:41:15.944509 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.944469    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"]
Nov 03 01:41:15.944987 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.944972    4550 cpu_manager.go:414] "containerMap: removing container" podUID="61d418fc-d568-4403-8fb4-7d98651ebdff" containerName="revoke-keys"
Nov 03 01:41:15.945048 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.944989    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d418fc-d568-4403-8fb4-7d98651ebdff" containerName="revoke-keys"
Nov 03 01:41:15.945112 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.945101    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d418fc-d568-4403-8fb4-7d98651ebdff" containerName="revoke-keys"
Nov 03 01:41:15.948648 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.948626    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:15.970082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.970047    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"]
Nov 03 01:41:15.977674 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.977636    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:41:15.977837 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.977739    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:41:15.992152 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.992113    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfxp\" (UniqueName: \"kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:15.992338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.992237    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:15.992338 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:15.992268    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.092759 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.092721    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfxp\" (UniqueName: \"kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.092954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.092814    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.092954 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.092837    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.093606 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.093587    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.095327 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.095292    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.113004 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.112971    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfxp\" (UniqueName: \"kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp\") pod \"maas-api-revoke-sub-f761c6d1-9nlb2\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.264901 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.264809    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:16.559227 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:41:16.559178    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd0efe0_5a58_4f52_8ddb_8e24b18061d1.slice/crio-730dc90e602d8b5765b4fccabc387bba1df3db7c2276769efda69be0bc4f067f WatchSource:0}: Error finding container 730dc90e602d8b5765b4fccabc387bba1df3db7c2276769efda69be0bc4f067f: Status 404 returned error can't find the container with id 730dc90e602d8b5765b4fccabc387bba1df3db7c2276769efda69be0bc4f067f
Nov 03 01:41:16.559618 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.559580    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"]
Nov 03 01:41:16.648335 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:16.648293    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2" event={"ID":"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1","Type":"ContainerStarted","Data":"730dc90e602d8b5765b4fccabc387bba1df3db7c2276769efda69be0bc4f067f"}
Nov 03 01:41:17.659222 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:17.659174    4550 generic.go:356] "Generic (PLEG): container finished" podID="ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" containerID="c7702ea9eed4a275cd4b2226dce2cc9d795931d4403099b19c23a06adc2bc029" exitCode=0
Nov 03 01:41:17.659782 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:17.659231    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2" event={"ID":"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1","Type":"ContainerDied","Data":"c7702ea9eed4a275cd4b2226dce2cc9d795931d4403099b19c23a06adc2bc029"}
Nov 03 01:41:19.145445 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.145399    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:19.165464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.161363    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:41:19.335233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.335200    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca\") pod \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") "
Nov 03 01:41:19.335552 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.335526    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca\") pod \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") "
Nov 03 01:41:19.335717 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.335688    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca" pod "ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" (UID: "ecd0efe0-5a58-4f52-8ddb-8e24b18061d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:19.335802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.335724    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp\" (UniqueName: \"kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp\") pod \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\" (UID: \"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1\") "
Nov 03 01:41:19.336024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.335998    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:19.337927 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.337860    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca" pod "ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" (UID: "ecd0efe0-5a58-4f52-8ddb-8e24b18061d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:19.338824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.338803    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp" pod "ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" (UID: "ecd0efe0-5a58-4f52-8ddb-8e24b18061d1"). InnerVolumeSpecName "kube-api-access-txfxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:19.437020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.436967    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfxp\" (UniqueName: \"kubernetes.io/projected/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-kube-api-access-txfxp\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:19.437020 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.437017    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd0efe0-5a58-4f52-8ddb-8e24b18061d1-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:19.713461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.712808    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2" event={"ID":"ecd0efe0-5a58-4f52-8ddb-8e24b18061d1","Type":"ContainerDied","Data":"730dc90e602d8b5765b4fccabc387bba1df3db7c2276769efda69be0bc4f067f"}
Nov 03 01:41:19.713461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.712875    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="730dc90e602d8b5765b4fccabc387bba1df3db7c2276769efda69be0bc4f067f"
Nov 03 01:41:19.713461 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:19.712962    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f761c6d1-9nlb2"
Nov 03 01:41:24.190955 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:24.190916    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:41:24.723342 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:41:24.723306    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:41:26.392121 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.392081    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"]
Nov 03 01:41:26.392547 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.392481    4550 cpu_manager.go:414] "containerMap: removing container" podUID="ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" containerName="revoke-keys"
Nov 03 01:41:26.392547 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.392494    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" containerName="revoke-keys"
Nov 03 01:41:26.392621 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.392585    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd0efe0-5a58-4f52-8ddb-8e24b18061d1" containerName="revoke-keys"
Nov 03 01:41:26.396008 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.395988    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.414162 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.413506    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"]
Nov 03 01:41:26.422798 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.422767    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 01:41:26.422945 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.422819    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9pms7\""
Nov 03 01:41:26.500061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.500013    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxrm\" (UniqueName: \"kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.500061 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.500066    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.500304 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.500225    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.601136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.601092    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxrm\" (UniqueName: \"kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.601136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.601135    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.601464 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.601179    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.602017 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.601996    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.603734 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.603712    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.621093 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.621056    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxrm\" (UniqueName: \"kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm\") pod \"maas-api-revoke-sub-197fbcc4-5vvft\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:26.709766 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:26.709730    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:27.009480 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:41:27.009418    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8577a9df_eaec_4f97_a982_4d087f752854.slice/crio-ca4d261a9bb409473714be5238c832a1a61866b3fe2d88f8f6c8f6ab749d8c5d WatchSource:0}: Error finding container ca4d261a9bb409473714be5238c832a1a61866b3fe2d88f8f6c8f6ab749d8c5d: Status 404 returned error can't find the container with id ca4d261a9bb409473714be5238c832a1a61866b3fe2d88f8f6c8f6ab749d8c5d
Nov 03 01:41:27.010744 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:27.010718    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"]
Nov 03 01:41:27.776663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:27.768009    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft" event={"ID":"8577a9df-eaec-4f97-a982-4d087f752854","Type":"ContainerStarted","Data":"f04eb9718a5f72189fa42ce6f449fe0affc4520cf8e3b3ff2c71d011a292b67b"}
Nov 03 01:41:27.776663 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:27.768083    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft" event={"ID":"8577a9df-eaec-4f97-a982-4d087f752854","Type":"ContainerStarted","Data":"ca4d261a9bb409473714be5238c832a1a61866b3fe2d88f8f6c8f6ab749d8c5d"}
Nov 03 01:41:27.840036 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:27.839965    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft" podStartSLOduration=1.839943681 podStartE2EDuration="1.839943681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:27.824693558 +0000 UTC m=+5770.146230358" watchObservedRunningTime="2026-11-03 01:41:27.839943681 +0000 UTC m=+5770.161480482"
Nov 03 01:41:28.787393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:28.786859    4550 generic.go:356] "Generic (PLEG): container finished" podID="8577a9df-eaec-4f97-a982-4d087f752854" containerID="f04eb9718a5f72189fa42ce6f449fe0affc4520cf8e3b3ff2c71d011a292b67b" exitCode=0
Nov 03 01:41:28.787393 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:28.786932    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft" event={"ID":"8577a9df-eaec-4f97-a982-4d087f752854","Type":"ContainerDied","Data":"f04eb9718a5f72189fa42ce6f449fe0affc4520cf8e3b3ff2c71d011a292b67b"}
Nov 03 01:41:29.933082 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:29.933050    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:30.048356 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.048267    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca\") pod \"8577a9df-eaec-4f97-a982-4d087f752854\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") "
Nov 03 01:41:30.048546 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.048375    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm\" (UniqueName: \"kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm\") pod \"8577a9df-eaec-4f97-a982-4d087f752854\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") "
Nov 03 01:41:30.048546 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.048416    4550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca\") pod \"8577a9df-eaec-4f97-a982-4d087f752854\" (UID: \"8577a9df-eaec-4f97-a982-4d087f752854\") "
Nov 03 01:41:30.048891 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.048860    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca" pod "8577a9df-eaec-4f97-a982-4d087f752854" (UID: "8577a9df-eaec-4f97-a982-4d087f752854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:41:30.050380 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.050333    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca" pod "8577a9df-eaec-4f97-a982-4d087f752854" (UID: "8577a9df-eaec-4f97-a982-4d087f752854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:41:30.050520 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.050411    4550 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm" pod "8577a9df-eaec-4f97-a982-4d087f752854" (UID: "8577a9df-eaec-4f97-a982-4d087f752854"). InnerVolumeSpecName "kube-api-access-2hxrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:41:30.149414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.149366    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-openshift-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:30.149414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.149409    4550 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8577a9df-eaec-4f97-a982-4d087f752854-maas-api-xks-ca\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:30.149414 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.149420    4550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxrm\" (UniqueName: \"kubernetes.io/projected/8577a9df-eaec-4f97-a982-4d087f752854-kube-api-access-2hxrm\") on node \"ip-10-0-133-253.ec2.internal\" DevicePath \"\""
Nov 03 01:41:30.797654 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.797620    4550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft"
Nov 03 01:41:30.797853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.797657    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-197fbcc4-5vvft" event={"ID":"8577a9df-eaec-4f97-a982-4d087f752854","Type":"ContainerDied","Data":"ca4d261a9bb409473714be5238c832a1a61866b3fe2d88f8f6c8f6ab749d8c5d"}
Nov 03 01:41:30.797853 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:30.797735    4550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca4d261a9bb409473714be5238c832a1a61866b3fe2d88f8f6c8f6ab749d8c5d"
Nov 03 01:41:54.726541 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:41:54.726497    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:41:57.027560 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:57.027525    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"]
Nov 03 01:41:57.040812 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:57.040775    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-218ade40-wzn6p"]
Nov 03 01:41:57.806870 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:41:57.806833    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b8d459-dd3f-4cb9-812b-82312bfe7607" path="/var/lib/kubelet/pods/77b8d459-dd3f-4cb9-812b-82312bfe7607/volumes"
Nov 03 01:42:01.148715 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:01.148630    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2hwfp_8588e986-19d2-4ce9-9c19-fc2ae2859e26/manager/0.log"
Nov 03 01:42:01.345595 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:01.345556    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lrpvq_5a235b0f-ffbd-4cb2-bc29-19ade07bb186/manager/0.log"
Nov 03 01:42:01.564976 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:01.564942    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p876r_1de0029a-6ab4-47db-9562-805aab89a6ea/manager/0.log"
Nov 03 01:42:02.081970 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:02.081939    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hfkg4_4c24118b-7393-4ad0-96ae-11d71d2dad5b/server/0.log"
Nov 03 01:42:03.836710 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:03.836681    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-197fbcc4-5vvft_8577a9df-eaec-4f97-a982-4d087f752854/revoke-keys/0.log"
Nov 03 01:42:04.021110 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:04.021067    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3748db8a-kfnpv_937ffed2-8cc8-4eb1-a75a-0f7f1e8da76f/revoke-keys/0.log"
Nov 03 01:42:04.212089 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:04.212055    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-574e406d-grh79_f4db3884-6598-45f1-ab96-d86816dd4a95/revoke-keys/0.log"
Nov 03 01:42:04.400139 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:04.400109    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-64c806ff-hkmnw_74164494-9b5d-496e-84a0-369a51087d1a/revoke-keys/0.log"
Nov 03 01:42:04.580277 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:04.580198    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7c759a18-6n4hd_a7ed6a4f-2201-46f7-aea3-930a0cbd4bd6/revoke-keys/0.log"
Nov 03 01:42:04.749228 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:04.749200    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8c1aaced-8nfnt_a1b711c1-11cd-467c-b474-273f22eb0cad/revoke-keys/0.log"
Nov 03 01:42:04.916088 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:04.916044    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a51b5ee1-d7kzm_61d418fc-d568-4403-8fb4-7d98651ebdff/revoke-keys/0.log"
Nov 03 01:42:05.106105 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:05.106071    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aeb7e7e4-6clw6_815949c5-ac36-4022-aa58-1fbd37b6df9c/revoke-keys/0.log"
Nov 03 01:42:05.298160 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:05.298079    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bf03952d-spjdd_2f5b8e45-58d2-4607-9041-6ae81abf1715/revoke-keys/0.log"
Nov 03 01:42:05.481792 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:05.481760    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c2fd3996-2jwgf_fbd904cf-530a-4530-ab31-d690451bbcdc/revoke-keys/0.log"
Nov 03 01:42:05.663334 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:05.663301    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3cc145c-kpz9l_27b5a37a-40c4-4f8b-917d-383d4cab4c3f/revoke-keys/0.log"
Nov 03 01:42:05.839986 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:05.839947    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d755f8d3-5ltgq_82b483c5-2338-4cbf-b399-9fcd344b3959/revoke-keys/0.log"
Nov 03 01:42:06.024749 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:06.024665    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e28ec5c3-dlc6p_bcda6729-715b-460f-889c-1993d9ab6570/revoke-keys/0.log"
Nov 03 01:42:06.215136 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:06.215105    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f0877206-zsfjb_978a4fec-a879-49ed-81e7-b1127cebbde6/revoke-keys/0.log"
Nov 03 01:42:06.397386 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:06.397314    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f0d766b6-6bvvx_b6728ce4-0e1f-4b9f-a234-3fdaab0bfe72/revoke-keys/0.log"
Nov 03 01:42:06.579529 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:06.579500    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f4cd3968-pp72g_64ffcfd7-1b38-4895-8879-71b75efc1ab7/revoke-keys/0.log"
Nov 03 01:42:06.758341 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:06.758305    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f761c6d1-9nlb2_ecd0efe0-5a58-4f52-8ddb-8e24b18061d1/revoke-keys/0.log"
Nov 03 01:42:06.948686 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:06.948655    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fcab913e-zkwjx_168151b5-24de-42c3-bc11-5a08d6a252af/revoke-keys/0.log"
Nov 03 01:42:07.038943 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:07.038844    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"]
Nov 03 01:42:07.045078 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:07.045043    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1aaced-8nfnt"]
Nov 03 01:42:07.130849 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:07.130810    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-mr4ms_9ce8b481-b56a-48a3-a84c-9bf39b36d2ff/postgres/0.log"
Nov 03 01:42:07.806063 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:07.806026    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b711c1-11cd-467c-b474-273f22eb0cad" path="/var/lib/kubelet/pods/a1b711c1-11cd-467c-b474-273f22eb0cad/volumes"
Nov 03 01:42:09.634643 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:09.634612    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jk4jg_624eee0c-d8af-4382-876c-f8ddac59b3cf/kuadrant-console-plugin/0.log"
Nov 03 01:42:09.891893 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:09.891799    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f4r7b_3b3a7806-8c9f-4e02-9848-cc488d0622ba/manager/0.log"
Nov 03 01:42:12.247657 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:12.247623    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-44ncj_f174f791-f438-4076-a30c-ecaca62452e5/istio-proxy/0.log"
Nov 03 01:42:12.872252 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:12.872224    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-gknbx_f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71/istio-proxy/0.log"
Nov 03 01:42:13.652822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:13.652793    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-rcjbb_caac059c-fc5f-4b53-99d5-c98bccf3480d/istio-proxy/0.log"
Nov 03 01:42:19.268964 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:19.268919    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:42:24.298467 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:24.298409    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:42:24.722107 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:42:24.722068    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:42:27.856930 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:27.856901    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:42:27.951757 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:27.951729    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:42:31.211339 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:31.211298    4550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log" limit=1024
Nov 03 01:42:33.109128 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:33.109097    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9rxv7_6a523c36-7d66-4549-b5fd-a9bc4903f53f/global-pull-secret-syncer/0.log"
Nov 03 01:42:34.060442 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:34.060407    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4zf7k_dcc07a6d-27f1-4b19-85e0-bbb1012fa1b3/konnectivity-agent/0.log"
Nov 03 01:42:34.614038 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:34.614004    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-253.ec2.internal_e107905b7a9295ebc233864ba55f6a44/haproxy/0.log"
Nov 03 01:42:40.062700 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:40.062656    4550 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"]
Nov 03 01:42:40.074976 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:40.074943    4550 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ec5c3-dlc6p"]
Nov 03 01:42:41.805880 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:41.805834    4550 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcda6729-715b-460f-889c-1993d9ab6570" path="/var/lib/kubelet/pods/bcda6729-715b-460f-889c-1993d9ab6570/volumes"
Nov 03 01:42:42.085075 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:42.084984    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jk4jg_624eee0c-d8af-4382-876c-f8ddac59b3cf/kuadrant-console-plugin/0.log"
Nov 03 01:42:42.286449 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:42.286406    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f4r7b_3b3a7806-8c9f-4e02-9848-cc488d0622ba/manager/0.log"
Nov 03 01:42:43.333944 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:43.333910    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2hwfp_8588e986-19d2-4ce9-9c19-fc2ae2859e26/manager/0.log"
Nov 03 01:42:43.504400 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:43.504371    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lrpvq_5a235b0f-ffbd-4cb2-bc29-19ade07bb186/manager/0.log"
Nov 03 01:42:44.176841 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.176793    4550 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"]
Nov 03 01:42:44.177374 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.177351    4550 cpu_manager.go:414] "containerMap: removing container" podUID="8577a9df-eaec-4f97-a982-4d087f752854" containerName="revoke-keys"
Nov 03 01:42:44.177586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.177384    4550 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8577a9df-eaec-4f97-a982-4d087f752854" containerName="revoke-keys"
Nov 03 01:42:44.177586 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.177526    4550 memory_manager.go:365] "RemoveStaleState removing state" podUID="8577a9df-eaec-4f97-a982-4d087f752854" containerName="revoke-keys"
Nov 03 01:42:44.181378 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.181353    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.199684 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.199651    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"]
Nov 03 01:42:44.225603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.225541    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-lib-modules\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.225603 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.225606    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zclvz\" (UniqueName: \"kubernetes.io/projected/32321bca-b6b9-49de-b60e-8d5f54e26a37-kube-api-access-zclvz\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.225911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.225729    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-podres\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.225911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.225775    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-sys\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.225911 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.225885    4550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-proc\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.283624 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.283548    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n428r\"/\"openshift-service-ca.crt\""
Nov 03 01:42:44.283822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.283645    4550 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n428r\"/\"default-dockercfg-dmzvq\""
Nov 03 01:42:44.283822 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.283642    4550 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n428r\"/\"kube-root-ca.crt\""
Nov 03 01:42:44.327024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.326992    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-podres\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327024 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327026    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-sys\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327129    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-proc\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327133    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-podres\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327177    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-sys\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327251 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327198    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-proc\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327255    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-lib-modules\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327277    4550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zclvz\" (UniqueName: \"kubernetes.io/projected/32321bca-b6b9-49de-b60e-8d5f54e26a37-kube-api-access-zclvz\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.327395 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.327360    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32321bca-b6b9-49de-b60e-8d5f54e26a37-lib-modules\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.348519 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.348381    4550 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zclvz\" (UniqueName: \"kubernetes.io/projected/32321bca-b6b9-49de-b60e-8d5f54e26a37-kube-api-access-zclvz\") pod \"perf-node-gather-daemonset-759x9\" (UID: \"32321bca-b6b9-49de-b60e-8d5f54e26a37\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.500679 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.500586    4550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:44.810064 ip-10-0-133-253 kubenswrapper[4550]: W1103 01:42:44.810024    4550 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32321bca_b6b9_49de_b60e_8d5f54e26a37.slice/crio-c9da27abb038de7f802f32e6eb64dd6917aac2668f1209b57fccedee314aa634 WatchSource:0}: Error finding container c9da27abb038de7f802f32e6eb64dd6917aac2668f1209b57fccedee314aa634: Status 404 returned error can't find the container with id c9da27abb038de7f802f32e6eb64dd6917aac2668f1209b57fccedee314aa634
Nov 03 01:42:44.811271 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.811033    4550 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"]
Nov 03 01:42:44.884805 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:44.884771    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p876r_1de0029a-6ab4-47db-9562-805aab89a6ea/manager/0.log"
Nov 03 01:42:45.184383 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:45.184345    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9" event={"ID":"32321bca-b6b9-49de-b60e-8d5f54e26a37","Type":"ContainerStarted","Data":"d6ac9e12e9eaa2a062cd15f336ec85ac0f1fa2e74854e0b0e938a76d1a1a03c1"}
Nov 03 01:42:45.184602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:45.184398    4550 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9" event={"ID":"32321bca-b6b9-49de-b60e-8d5f54e26a37","Type":"ContainerStarted","Data":"c9da27abb038de7f802f32e6eb64dd6917aac2668f1209b57fccedee314aa634"}
Nov 03 01:42:45.184602 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:45.184457    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:45.225272 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:45.225213    4550 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9" podStartSLOduration=1.22519976 podStartE2EDuration="1.22519976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:45.22355305 +0000 UTC m=+5847.545089850" watchObservedRunningTime="2026-11-03 01:42:45.22519976 +0000 UTC m=+5847.546736561"
Nov 03 01:42:45.349119 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:45.349088    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hfkg4_4c24118b-7393-4ad0-96ae-11d71d2dad5b/server/0.log"
Nov 03 01:42:45.890475 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:45.890426    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66psj_2fa98c9b-3ffb-4806-8ba0-8350e96b7517/node-exporter/0.log"
Nov 03 01:42:46.010824 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:46.010792    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66psj_2fa98c9b-3ffb-4806-8ba0-8350e96b7517/kube-rbac-proxy/0.log"
Nov 03 01:42:46.132985 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:46.132957    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-66psj_2fa98c9b-3ffb-4806-8ba0-8350e96b7517/init-textfile/0.log"
Nov 03 01:42:47.038845 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.038815    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2hwfp_8588e986-19d2-4ce9-9c19-fc2ae2859e26/manager/0.log"
Nov 03 01:42:47.204946 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.204907    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lrpvq_5a235b0f-ffbd-4cb2-bc29-19ade07bb186/manager/0.log"
Nov 03 01:42:47.445183 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.445145    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/prometheus/0.log"
Nov 03 01:42:47.570296 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.570259    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/config-reloader/0.log"
Nov 03 01:42:47.698558 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.698400    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/thanos-sidecar/0.log"
Nov 03 01:42:47.824179 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.824150    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/kube-rbac-proxy-web/0.log"
Nov 03 01:42:47.951053 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:47.950962    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/kube-rbac-proxy/0.log"
Nov 03 01:42:48.077866 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:48.077832    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/kube-rbac-proxy-thanos/0.log"
Nov 03 01:42:48.209307 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:48.209211    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_661ba1e5-59ed-468f-a9ce-41483779bfcb/init-config-reloader/0.log"
Nov 03 01:42:48.493482 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:48.493365    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p876r_1de0029a-6ab4-47db-9562-805aab89a6ea/manager/0.log"
Nov 03 01:42:48.592729 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:48.592691    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-tqj4d_c9610df3-39ba-4b02-8daa-71ec7facb86f/prometheus-operator-admission-webhook/0.log"
Nov 03 01:42:48.722885 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:48.722856    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hfkg4_4c24118b-7393-4ad0-96ae-11d71d2dad5b/server/0.log"
Nov 03 01:42:50.271233 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:50.271206    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-2hwfp_8588e986-19d2-4ce9-9c19-fc2ae2859e26/manager/0.log"
Nov 03 01:42:50.438041 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:50.437990    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-lrpvq_5a235b0f-ffbd-4cb2-bc29-19ade07bb186/manager/0.log"
Nov 03 01:42:51.201178 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:51.201145    4550 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-759x9"
Nov 03 01:42:51.718615 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:51.718574    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-p876r_1de0029a-6ab4-47db-9562-805aab89a6ea/manager/0.log"
Nov 03 01:42:51.953754 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:51.953725    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-hfkg4_4c24118b-7393-4ad0-96ae-11d71d2dad5b/server/0.log"
Nov 03 01:42:53.750316 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:53.750274    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-tm57s_7e1bf19b-8b56-488d-bf10-9e3170b23393/download-server/0.log"
Nov 03 01:42:54.315802 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:54.315759    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-44ncj_f174f791-f438-4076-a30c-ecaca62452e5/istio-proxy/0.log"
Nov 03 01:42:54.548007 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:54.547978    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-gknbx_f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71/istio-proxy/0.log"
Nov 03 01:42:54.723488 ip-10-0-133-253 kubenswrapper[4550]: E1103 01:42:54.723451    4550 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4cnb6" containerName="extractor"
Nov 03 01:42:54.836371 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:54.836338    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-rcjbb_caac059c-fc5f-4b53-99d5-c98bccf3480d/istio-proxy/0.log"
Nov 03 01:42:57.788719 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:57.788680    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pzqlp_24c625e0-13a7-48df-bd0e-c383ff56ecca/dns/0.log"
Nov 03 01:42:57.905806 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:57.905766    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pzqlp_24c625e0-13a7-48df-bd0e-c383ff56ecca/kube-rbac-proxy/0.log"
Nov 03 01:42:58.284869 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:58.284833    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jkfg_a91bc30d-2c15-48ab-97c9-c54713750b36/dns-node-resolver/0.log"
Nov 03 01:42:58.483372 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:58.483339    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-44ncj_f174f791-f438-4076-a30c-ecaca62452e5/istio-proxy/0.log"
Nov 03 01:42:58.682952 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:58.682911    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-gknbx_f89e0826-6db2-4e0d-8c06-e3e9ad3bdf71/istio-proxy/0.log"
Nov 03 01:42:58.959727 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:58.959635    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-rcjbb_caac059c-fc5f-4b53-99d5-c98bccf3480d/istio-proxy/0.log"
Nov 03 01:42:59.868227 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:59.868191    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57dc984b8f-b8bms_a185d8a1-09e2-464d-b73b-cc11e1b5a068/registry/0.log"
Nov 03 01:42:59.978117 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:42:59.978088    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-96xx2_887b3093-6795-4b74-aa85-f46f3dbc76fd/node-ca/0.log"
Nov 03 01:43:00.119536 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:43:00.119416    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-jk4jg_624eee0c-d8af-4382-876c-f8ddac59b3cf/kuadrant-console-plugin/0.log"
Nov 03 01:43:00.230498 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:43:00.230463    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-f4r7b_3b3a7806-8c9f-4e02-9848-cc488d0622ba/manager/0.log"
Nov 03 01:43:03.402533 ip-10-0-133-253 kubenswrapper[4550]: I1103 01:43:03.402495    4550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-44ncj_f174f791-f438-4076-a30c-ecaca62452e5/istio-proxy/0.log"